DETAILED ACTION
Status of Claims
The present application, filed on or after 3/16/2013, is being examined under the first inventor to file provisions of the AIA .
This action is in reply to the Remarks and Amendments filed 006/26/2025.
Claims 1, 12, 19 have been amended.
Claims 1-20 have been examined and are pending.
(AIA ) Examiner Note
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned at the time any inventions covered therein were effectively filed absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned at the time a later invention was effectively filed in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention
Duty to disclose information material to patentability
37 CFR 1.56 (a) A patent by its very nature is affected with a public interest. The public interest is best served, and the most effective patent examination occurs when, at the time an application is being examined, the Office is aware of and evaluates the teachings of all information material to patentability. Each individual associated with the filing and prosecution of a patent application has a duty of candor and good faith in dealing with the Office, which includes a duty to disclose to the Office all information known to that individual to be material to patentability as defined in this section. The duty to disclose information exists with respect to each pending claim until the claim is cancelled or withdrawn from consideration, or the application becomes abandoned. Information material to the patentability of a claim that is cancelled or withdrawn from consideration need not be submitted if the information is not material to the patentability of any claim remaining under consideration in the application. There is no duty to submit information which is not material to the patentability of any existing claim. The duty to disclose all information known to be material to patentability is deemed to be satisfied if all information known to be material to patentability of any claim issued in a patent was cited by the Office or submitted to the Office in the manner prescribed by §§ 1.97(b)-(d) and 1.98. However, no patent will be granted on an application in connection with which fraud on the Office was practiced or attempted or the duty of disclosure was violated through bad faith or intentional misconduct. The Office encourages applicants to carefully examine:
(1) Prior art cited in search reports of a foreign patent office in a counterpart application, and
(2) The closest information over which individuals associated with the filing or prosecution of a patent application believe any pending claim patentably defines, to make sure that any material information contained therein is disclosed to the Office.
37 CFR 1.56 (c) Individuals associated with the filing or prosecution of a patent application within the meaning of this section are:
(1) Each inventor named in the application;
(2) Each attorney or agent who prepares or prosecutes the application; and
(3) Every other person who is substantively involved in the preparation or prosecution of the application and who is associated with the inventor, with the assignee or with anyone to whom there is an obligation to assign the application.
Claim Rejections - 35 USC § 103 (AIA )
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 1-5, 7, 12, 14-15, 17, 19, 20 are rejected under 35 U.S.C. 103 as obvious over Costello et al. (U.S. 2019/0303676 A1; hereinafter, "Costello") in view of Chen et al. (U.S. 2019/0130583 A1; hereinafter, "Chen") in view of Fail et al (U.S. 2007/0112572 A1; hereinafter, "Fail").
Claim 1: (Currently amended)
Pertaining to claim 1, as shown, Costello teaches the following:
A method comprising:
capturing time-stamped videos of patrons within an establishment using a plurality of
cameras (Costello, see at least Fig. 3, [0030], and [0064]-[0074]. E.g. per [0030]: “It is noted that the metadata collected for the processed images from the video captured during the customer's visit includes data and time stamps and camera identifiers for each of the cameras 120 and 130 that created, added to, or deleted from the metadata…”);
authenticating a certain patron to an identity that is linked to a registered profile comprising a registered payment method based on biometric attributes captured from the time stamped videos (Costello, see citations noted supra, including again at least [0064]-[0074], e.g.: “…At 310, the real-time video tracker provides a unique identifier that is to be associated with a customer being tracked by decentralized video tracking agents within a store when a customer enters the store… the real-time video tracker identifies a customer identifier for the customer based on an automated check-in mechanism… (…automatically performed through
facial recognition) when the customer enters the store… Then, the real-time video tracker accesses a customer database using the customer identifier (which was mapped by the real-time video tracker to the unique identifier and in an embodiment where the customer is registered with the store) to obtain a customer record that includes the customer's registered payment method for the transaction. Third-party payment services may then be used to complete and verify the payment processing for the transaction. In some cases, based on a customer profile associated with the customer record…”);
tracking the certain patron to a location within the establishment using video analytics that employ […] patron identifiers associated with or assigned to pixels of the time-stamped videos (Costello, see citations noted supra, e.g. per Fig. 3 and at least [0064]-[0074]: “…At 310, the real-time video tracker provides a unique identifier [patron identifier] that is to be associated with a customer [patron] being tracked [tracked location] by decentralized video tracking agents within a store when a customer enters the store… The real-time video tracker links the customer identifier to the provided unique identifier… The identifiers allow the real-time video tracker to determine the physical locations of cameras for each of the tracking agents, the field-of-views for each tracking agent, and item attributes (including coordinates) for the real-time video tracker to identify items taken by the customer while in the store based on the unique identifier, the identifiers, and the metadata…”);
generating a service request for the certain patron based on at least one action, behavior, or gesture detected from the certain patron in the time-stamped videos (Costello, see citations noted supra, e.g. again per [0071]-[0074], teaching: “…At 320, the real-time video tracker receives metadata representing items taken [at least one action, behavior, or gesture detected] by the customer [from the certain patron] within the store and including the unique identifier... At 321…, once the items taken are identified, the real-time video tracker interacts with a store item database to obtain item descriptions and item pricing… At 330, the real-time video tracker processes an automated action responsive to evaluation of the metadata performed by the real-time video tracker (as discussed above in 320 and 321)…”; Costello per [0074] teaches that one such automated action may be to process a purchase transaction based on his “evaluation of the metadata”; applicant’s “service request” is understood to be a digital piece of information representing an evaluation or inference, by a computer algorithm, that a user has made a request for some type of service, e.g. made a request to purchase an item. Thus applicant’s “service request” reads on the evaluation and inference made by Costello’s system necessary to execute his “automated action” at step 330; i.e. Costello evaluates metadata and infers his customer is requesting a service to purchase the items which the customer picked up [an action, behavior, or gesture of the user and detected from the video] based on the metadata including item identifiers, etc…. Costello makes this evaluation and inference regarding his customer’s request and then automatically executes his “automated action” which may be processing a purchase transaction for the items which the user signaled he wanted to purchase via his action of picking up such items as detected in the video.);
[…]
processing a payment for the service item using the registered payment method (Costello, see citations noted supra, e.g. per [0074]: “…at 331, the real-time video tracker reconstructs an entire transaction for the items taken and the customer and completes payment for the transaction based on a registered payment method associated with the customer. That is, once the items taken are identified, the real-time video tracker interacts with a store item database to obtain item descriptions and item pricing and generates a receipt and a total price for the transaction…”) when the certain patron exits the establishment as determined by the video analytics (Costello, see citations noted supra in view of at least [0061]-[0062], teaching e.g.: “…At some point, the object (customer) is detected as approaching an egress point of the store [an indication, as determined by the video analytics, the certain patron exits the establishment]. For purposes of illustration this is identified by the further processing instance, in an embodiment of 271, at 272. The further processing instance creates final metadata from the modified metadata when the object (customer) is detected at or proximate to an exit point of the store within the further (third) video frames…”; the difference between the claims and the prior art of record is only that Costello appears to teach that his processing (i.e. of a payment for an item [a service item] taken by his customer, where the payment uses a registered payment method) occurs either when customer takes the item or approaches the egress/exit but not explicitly “when” the customer exits. However, examiner finds that there is a need to make a determination that a customer is exiting if a customer comes within some very close proximity of an egress and it is within the level of skill of a person of ordinary skill in the art to establish some proximity to an exit as an indication of a high-probability that a customer is existing or is in the process of exiting relative to the time at which the system may be processing such information and therefore, there is motivation to modify Costello’s teaching to process his payment transaction when the certain patron exits the establishment as determined by the video analytics because per MPEP 2143(I) (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference teachings to arrive at the claimed invention is obvious. The motivation may be implicit and may be found in the knowledge of one of ordinary skill in the art, or, in some cases, from the nature of the problem to be solved. Id. at 1366, 80 USPQ2d at 1649.)
monitoring a service item […] using object tracking that employs service item identifiers […] (Costello, see citations noted supra in view of at least Fig. 2, [0032]- [0035], and [047]-[0049], e.g.: per [0032]:”…The image tracking/transaction manager 141 is networked to the store's… item database (which includes each item's aisle location and shelf location within that aisle). The image tracking/transaction manager 141 processes the metadata… to search the item database for specific items…”; per [0057]: “…at 270, the next processing instance of the decentralized video agent modifies the metadata to include item attributes for an item detected as being picked up by the object within the next video frames being captured…”; per [0072]: “…The identifiers allow the real-time video tracker to determine the physical locations of cameras for each of the tracking agents, the field-of-views for each tracking agent, and item attributes (including coordinates) for the real-time video tracker to identify items taken by the customer while in the store based on the unique identifier, the identifiers, and the metadata….”);
Although Costello teaches the above limitations including real-time time-stamped video monitoring and tracking of the movement of customers and the items taken by such customer moving within a “frictionless store” environment where such customers do not need to directly interact with a service person to pay for items taken (e.g. including items being consumed), Costello may not explicitly contemplate the nuance recited below regarding his video and object tracking technology employing “bounding boxes” within the pixels of his time-stamped videos. However, regarding this nuanced feature, Costello in view of Chen teaches the following:
video analytics that employ bounding boxes; and object tracking that employs… the bounding boxes within the pixels of the time-stamped videos (Chen, see at least [0073], teaching, e.g.: “…Video analytics also provides various other features. For example, video analytics can operate as an Intelligent Video Motion Detector by detecting moving objects and by tracking moving objects. In some cases, the video analytics can generate and display a bounding box around a valid object. Video analytics can also act as an intrusion detector, a video counter (e.g., by counting people, objects, vehicles, or the like), a camera tamper detector, an object left detector, an object/asset removal detector, an asset protector, a loitering detector, and/or as a slip and fall detector. Video analytics can further be used to perform various types of recognition functions, such as face detection and recognition, license plate recognition, object recognition (e.g., bags, logos, body marks, or the like), or other recognition functions. In some cases, video analytics can be trained to recognize certain objects. Another function that can be performed by video analytics includes providing demographics for customer metrics (e.g., customer counts, gender, age, amount of time spent, and other suitable metrics)…” and per at least [0007]: “…An object classified by the complex object detector can be localized using a bounding region (e.g., a bounding box or other bounding region) representing the classified object….”)
Therefore, the Examiner understands that the limitation in question is merely applying a known technique of Chen (directed towards video analytics and object recognition which makes use of bounding boxes around valid objects for the purpose of object classification) which is applicable to a known base device/method of Costello (already directed towards using video analytics to identify people and objects, e.g. via facial recognition and similar techniques) to yield predictable results. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the techniques of Chen to the device/method of Costello to improve Costello’s person and object recognition system and method because Chen is pertinent to the video analytics and object recognition aspects of Costello and because according to MPEP 2143(I) (C) and/or (D), the use of known technique to improve a known device, methods, or products in the same way (or which is ready for improvement) is obvious.
Although Costello/Chen teaches the above limitations including monitoring and tracking the movement of customers, as well as monitoring and identifying items taken by such customer moving within a “frictionless store” environment where such customers do not need to directly interact with a service person to pay for items taken (e.g. including items being consumed), and Costello teaches he alerts service staff of activities which may require their interaction with a customer, e.g. such as potential theft as described at least per [0092], Costello may not explicitly contemplate the nuanced situation and method of monitoring a service item being prepared for a customer’s request nor of alerting/dispatching service staff to deliver such items when ready for delivery. However, regarding these features, Costello in view of Fail teaches the following:
[monitoring a service item] being prepared for the service request (Fail, see at least [0037]-[0038], teaching, e.g.: “…Once the food or other service item is ready for delivery to the individual, the waiter or other responsible individual is prompted via pager or other electronic device 210 to pick-up and deliver the food to the waiting individual…”; applicant’s “monitoring” reads on Fail’s determination that the service item is “ready”)
dispatching a service staff member to deliver the service item to the certain patron at the
location when the object tracking monitoring indicates the service item is ready for pickup by the service staff member (Fail, see again at least [0037]-[0038], teaching e.g.: “…Once the food or other service item is ready [when the object tracking monitoring indicates the service item is ready] for delivery to the individual, the waiter or other responsible individual is prompted [dispatching a service staff member] via pager or other electronic device 210 to pick-up and deliver the food [to deliver the service item] to the waiting individual [to the certain patron]…”)
Therefore, the Examiner understands that the limitations in question are merely applying known techniques of Fail (directed towards techniques of monitoring for the readiness of an item being prepared for a customer at a customer’s request) which are applicable to a known base device/method of Costello (already directed towards monitoring service items as well as customers via cameras to identify which items a customer takes or receives in order to electronically bill the customer in a “frictionless store” environment) to yield predictable results. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the techniques of Fail to the device/method of Costello such that Costello also provides this service per the technique of Fail to his customers to enhance the services provided by his “frictionless store” to his customers because Fail is pertinent to the customer service aspects of Costello and because according to MPEP 2143(I) (C) and/or (D), the use of known technique to improve a known device, methods, or products in the same way (or which is ready for improvement) is obvious.
Claim 2: (previously presented)
Costello/Chen/Fail teaches the limitations upon which this claim depends. Furthermore, Costello teaches the following:
The method of claim 1 further comprising, monitoring the certain patron, the service staff member, and one or more service preparation staff members within the time-stamped videos for prohibited behaviors or prohibited actions (Costello, see citations noted supra in view of at least [0034] and [0092], e.g. per [0034]: “…It is to be noted that the above-noted processing can be used for security purposes as well to detect and thwart theft [prohibited action]…”; and per [0092]: “…Still further, the video transaction manager 402 can process the action as a mechanism for raising alarms or sending notifications to store equipment when theft is suspected by the customer based on the metadata. In an embodiment, the video transaction manager 402 can process multiple actions for a frictionless-store transaction, verifying a transaction terminal transaction, and raising automated alarms or sending alerts for theft prevention [prohibited behaviors or actions]...”)
Claim 3: (previously presented)
Costello/Chen/Fail teaches the limitations upon which this claim depends. Furthermore, Costello teaches the following:
The method of claim 2 further comprising, capturing service item preparation metrics for the one or more service preparation staff members during preparation of the service item from the time-stamped videos and capturing service item delivery metrics for the
service staff member from the time-stamped videos (Costello, see at least [0027]-[0030], teaching: “…The object tracker 121 or 131 also expands on the metadata [metrics] when the customer removes an item from a shelf within the store 110 by providing the coordinates from the images where an item was removed from a shelf, dimensions of the item [metrics of the service item], color of the item [metrics of the service item], etc… The metadata can also reveal that a customer that drops an item at a different location within the store 110 may backtrack and re-pickup that same item from the different location…”; this teaching indicates Costello is collecting information, using his time-stamped videos, where such information includes metadata [metrics] as service items are picked-up and delivered by various persons to different areas of a store; also per [0071], teaching: “…At 320, the real-time video tracker receives metadata [metrics] representing items [service items] taken by the customer within the store and including the unique identifier. The metadata determined and produced by the decentralized video tracking agents through image processing performed by the decentralized video tracking agents…”; Examiner notes the only difference between the claims and the teachings of the prior art is the designation of the person interacting with the service item, i.e. Costello explicitly teach customer picking up and dropping off service items within a store and collecting metrics and information both during pickup and drop-off but not necessarily teaching the same information may be collected for his service staff member interactions with service items. However, elsewhere Costello, e.g. per [0092] teaches: “…In an embodiment, the video transaction manager 402 can process multiple actions for a frictionless-store transaction, verifying a transaction terminal transaction, and raising automated alarms or sending alerts for theft prevention.” Which provides motivation to monitor for theft and Examiner finds it would be within the level of skill of a person of ordinary skill in the art to recognize the same need extends to theft of items by service staff members. Therefore, there is motivation to monitor any person-item interaction, including that of service staff member interaction with service items, and collect the same information for service staff member interactions as Costello explicitly teaches he collects for service items picked up and dropped off by customers with a goal to meet Costello’s objective of detecting potential theft as extended to service staff members because per MPEP 2143(I) (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention is obvious. The motivation to combine may be implicit and may be found in the knowledge of one of ordinary skill in the art, or, in some cases, from the nature of the problem to be solved. Id. at 1366, 80 USPQ2d at 1649.)
Claim 4: (previously presented)
Costello/Chen/Fail teaches the limitations upon which this claim depends. Furthermore, Costello teaches the following:
The method of claim 1, wherein authenticating further includes continuously capturing biometric attributes for the certain patron from the time-stamped videos and providing the biometric attributes to an authenticator until the authenticator authenticates the certain patron to the identity from the biometric attributes (Costello, see citations noted supra in view of at least [0023], e.g.: “…The automated check-in mechanism can include a variety of techniques, such as: … 3) biometrics where the customer touches a fingerprint reader and the biometrics registered to the customer, 4) facial recognition where the customer's face image and face biometrics are registered with the server, etc….” and again per [0069]: “…at 311, the real-time video tracker identifies a customer identifier for the customer based on an automated check-in mechanism performed by the customer ( or automatically performed through a device carried by the customer or automatically performed through facial recognition) when the customer enters the store…”)
Claim 5: (previously presented)
Costello/Chen/Fail teaches the limitations upon which this claim depends. Furthermore, Costello teaches the following:
The method of claim 1, wherein authenticating further includes receiving an automated check-in message from a mobile device of the certain patron when the mobile device detects the certain patron within the establishment and authenticating the certain patron to the identity based on a mobile device identifier associated with the mobile device. (Costello, see citations noted supra in view of at least [0023], e.g.: “…The automated check-in mechanism can include a variety of techniques, such as: 1) the customer scanning a barcode (or Quick Response (QR) code) that triggers an application that executes on the mobile device to report a customer identifier for the customer to the server 140, 2) mobile device identification based on the mobile device reporting its physical location to the server 140 (such as through a mobile application executing on the mobile device) wherein the mobile device identifier for the mobile device is registered to the customer,….” and again per [0069]: “…at 311, the real-time video tracker identifies a customer identifier for the customer based on an automated check-in mechanism performed by the customer (or automatically performed through a device carried by the customer or automatically performed through facial recognition) when the customer enters the store…”)
Claim 7: (previously presented)
Costello/Chen/Fail teaches the limitations upon which this claim depends. Furthermore, Costello in view of Fail teaches the following:
The method of claim 1, wherein monitoring generating further includes sending a menu for the establishment to a mobile device operated by the certain patron and receiving the service request from selections of the certain patron with respect to the menu on the mobile device. (Fail, see citations noted supra in view of at least Figs. 1 and 7 and [0017]-[0033], e.g.: “…In the preferred embodiment, the present invention is a dedicated device 100 having the
capability to display information such as a menu to individuals with various degrees of vision impairment… Assuming that the individual desires pancakes for breakfast, they can either speak the word or select the pancake menu item using cursor 602 as illustrated by FIG. 7, one or more of the controls 102a-n, pressure sensitive display 104, or some combination thereof. An initial selection would be verified both audibly (indication of their selection and optionally the ability to verify with speech), if enabled, and with an additional step such as pressing an enter key or other input as previously explained…”)
Claim 12: (Currently Amended)
Pertaining to claim 12, as shown, Costello teaches the following:
A method comprising:
authenticating a customer within an establishment to a customer identity using a server that analyzes biometric features captured within time-stamped videos (Costello, see at least Fig. 3, [0030], and [0064]-[0074]. For example, per [0030]: “It is noted that the metadata collected for the processed images from the video captured during the customer's visit includes data and time stamps and camera identifiers for each of the cameras 120 and 130 that created, added to, or deleted from the metadata…” and per [0064]-[0074], e.g.: “…At 310, the real-time video tracker provides a unique identifier that is to be associated with a customer being tracked by decentralized video tracking agents within a store when a customer enters the store… the real-time video tracker identifies a customer identifier for the customer based on an automated check-in mechanism… (…automatically performed through facial recognition) when the customer enters the store… Then, the real-time video tracker accesses a customer database using the customer identifier (which was mapped by the real-time video tracker to the unique identifier and in an embodiment where the customer is registered with the store) to obtain a customer record that includes the customer's registered payment method for the transaction. Third-party payment services may then be used to complete and verify the payment processing for the transaction. In some cases, based on a customer profile associated with the customer record…”);
maintaining a current location of the customer within the establishment from the timestamped videos by employing video analytics that utilize […] and customer identifiers (Costello, see citations noted supra, e.g. [0068]-[0072]: “…At 310, the real-time video tracker provides a unique identifier [patron identifier] that is to be associated with a customer [patron] being tracked [tracked location] by decentralized video tracking agents within a store when a customer enters the store… The real-time video tracker links the customer identifier to the provided unique identifier… The identifiers allow the real-time video tracker to determine the physical locations of cameras for each of the tracking agents, the field-of-views for each tracking agent, and item attributes (including coordinates) for the real-time video tracker to identify items taken by the customer while in the store based on the unique identifier, the identifiers, and the metadata…”);
[…]
monitoring […] using the video analytics that employ […] staff identifiers (Costello, see citations noted supra in view of at least [0068]-[0075], e.g.: “…At 310, the real-time video tracker provides a unique identifier that is to be associated with a customer being tracked by decentralized video tracking agents within a store when a customer enters the store… The real-time video tracker sends an alert to staff devices [staff identifiers]”; applicant’s claim does not elaborate regarding employment of “staff identifiers” nor what they may entail and therefore applicant’s feature reads on Costello’s video analytics system which makes use of some identifier(s) associated with staff to enable his alerting of “staff devices”.);
Although Costello teaches the above limitations including real-time time-stamped video monitoring and tracking of the movement of customers and the items taken by such customer moving within a “frictionless store” environment where such customers do not need to directly interact with a service person to pay for items taken (e.g. including items being consumed), Costello may not explicitly contemplate the nuance recited below regarding his video and object tracking technology employing “bounding boxes” within the pixels of his time-stamped videos. However, regarding this nuanced feature, Costello in view of Chen teaches the following:
video analytics that utilize bounding boxes (Chen, see at least [0073], teaching, e.g.: “…Video analytics also provides various other features. For example, video analytics can operate as an Intelligent Video Motion Detector by detecting moving objects and by tracking moving objects. In some cases, the video analytics can generate and display a bounding box around a valid object. Video analytics can also act as an intrusion detector, a video counter (e.g., by counting people, objects, vehicles, or the like), a camera tamper detector, an object left detector, an object/asset removal detector, an asset protector, a loitering detector, and/or as a slip and fall detector. Video analytics can further be used to perform various types of recognition functions, such as face detection and recognition, license plate recognition, object recognition (e.g., bags, logos, body marks, or the like), or other recognition functions. In some cases, video analytics can be trained to recognize certain objects. Another function that can be performed by video analytics includes providing demographics for customer metrics (e.g., customer counts, gender, age, amount of time spent, and other suitable metrics)…” and per at least [0007]: “…An object classified by the complex object detector can be localized using a bounding region (e.g., a bounding box or other bounding region) representing the classified object….”)
Therefore, the Examiner understands that the limitation in question is merely applying a known technique of Chen (directed towards video analytics and object recognition which makes use of bounding boxes around valid objects for the purpose of object classification) which is applicable to a known base device/method of Costello (already directed towards using video analytics to identify people and objects, e.g. via facial recognition and similar techniques) to yield predictable results. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the techniques of Chen to the device/method of Costello to improve Costello’s person and object recognition system and method because Chen is pertinent to the video analytics and object recognition aspects of Costello and because according to MPEP 2143(I) (C) and/or (D), the use of known technique to improve a known device, methods, or products in the same way (or which is ready for improvement) is obvious.
Although Costello/Chen teaches the above limitations and they teach monitoring […] using the video analytics that employ the bounding boxes and staff identifiers as already shown supra, and they teach including monitoring and tracking the movement of customers, as well as monitoring and identifying items taken by such customer moving within a “frictionless store” environment where such customers do not need to directly interact with a service person to pay for items taken (e.g. including items being consumed), and Costello teaches he alerts service staff of activities which may require their interaction with a customer, e.g. such as potential theft as described at least per [0092], Costello may not explicitly contemplate the nuanced situation and method of monitoring a service item being prepared for a customer’s request nor of alerting/dispatching service staff to deliver such items when ready for delivery. However, regarding these features, Costello in view of Fail teaches the following:
receiving an order from a mobile device of the customer (Fail, see at least [0009], [0033]-[0036] and Figs. 1-7, Fail’s system receives a customer selection [order] via the customer mobile device
PNG
media_image1.png
468
522
media_image1.png
Greyscale
);
placing the order with service preparation staff (Fail, see again citations noted supra, and also again at least [0024], teaching e.g.: “…Workstation 208 can, for example, be located in the
kitchen or food preparation managing area of a restaurant. Workstation 208 can receive information, as described in connection with FIG. 3, directly from the dedicated device 100 or routed through the Server 202 for tracking orders, …”)
notifying a service staff member to deliver the order to the current location of the customer when the monitoring indicates the order is prepared and ready for pickup
and delivery to the customer (Fail, see again at least [0037]-[0038], teaching e.g.: “…Once the food or other service item is ready [when the object tracking monitoring indicates the service item is ready] for delivery to the individual, the waiter or other responsible individual is prompted [notifying/dispatching a service staff member] via pager or other electronic device 210 to pick-up and deliver the food [to deliver the service item] to the waiting individual [to the certain patron]…”)
Therefore, the Examiner understands that the limitations in question are merely applying known techniques of Fail (directed towards techniques of receiving customer orders, communicating the order with preparation staff, and then monitoring for the readiness of the item to be delivered to the customer per the customer’s request) which are applicable to a known base device/method of Costello/Chen (already directed towards monitoring service items as well as customers via cameras and video analytics using bounding boxes and unique identifiers associated with a person) to yield predictable results. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the techniques of Fail to the device/method of Costello such that Costello also provides this service per the technique of Fail to his customers to enhance the services provided by his “frictionless store” to his customers because Fail is pertinent to the customer service aspects of Costello and because according to MPEP 2143(I) (C) and/or (D), the use of known technique to improve a known device, methods, or products in the same way (or which is ready for improvement) is obvious.
Claim 14: (previously presented)
Costello/Chen/Fail teach the limitations upon which this claim depends. Furthermore, Costello teaches the following:
The method of claim 12 further comprising, automatically processing a payment for the order when the customer exits the establishment based on the video analytics. (Costello, see citations noted supra, e.g. per [0074]: “…at 331, the real-time video tracker reconstructs an entire transaction for the items taken and the customer and completes payment for the transaction based on a registered payment method associated with the customer. That is, once the items taken are identified, the real-time video tracker interacts with a store item database to obtain item descriptions and item pricing and generates a receipt and a total price for the transaction…” and per [0061]-[0062], teaching e.g.: “…At some point, the object (customer) is detected as approaching an egress point of the store [an indication, as determined by the video analytics, the certain patron exits the establishment]. For purposes of illustration this is identified by the further processing instance, in an embodiment of 271, at 272. The further processing instance creates final metadata from the modified metadata when the object (customer) is detected at or proximate to an exit point of the store within the further (third) video frames…”; the difference between the claims and the prior art of record is only that Costello appears to teach that his processing (i.e. of a payment for an item [a service item] taken by his customer, where the payment uses a registered payment method) occurs either when customer takes the item or approaches the egress/exit but not explicitly “when” the customer exits. However, examiner finds that there is a need to make a determination that a customer is exiting if a customer comes within some very close proximity of an egress and it is within the level of skill of a person of ordinary skill in the art to establish some proximity to an exit as an indication of a high-probability that a customer is existing or is in the process of exiting relative to the time at which the system may be processing such information and therefore, there is motivation to modify Costello’s teaching to process his payment transaction when the certain patron exits the establishment as determined by the video analytics because per MPEP 2143(I) (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference teachings to arrive at the claimed invention is obvious. The motivation may be implicit and may be found in the knowledge of one of ordinary skill in the art, or, in some cases, from the nature of the problem to be solved. Id. at 1366, 80 USPQ2d at 1649.)
Claim 15: (previously presented)
Costello/Chen/Fail teaches the limitations upon which this claim depends. Furthermore, Costello teaches the following:
The method of claim 12 further comprising, raising an alert to an automated system or to a designated resource based on certain video analytics indicating a prohibited action or a prohibited behavior by one or more of the service preparation staff, the service staff member, or the customer (Costello, see citations noted supra in view of at least [0034] and [0092], e.g. per [0034]: “…It is to be noted that the above-noted processing can be used for security purposes as well to detect and thwart theft [prohibited action]…”; and per [0092]: “…Still further, the video transaction manager 402 can process the action as a mechanism for raising alarms or sending notifications to store equipment when theft is suspected by the customer based on the metadata. In an embodiment, the video transaction manager 402 can process multiple actions for a frictionless-store transaction, verifying a transaction terminal transaction, and raising automated alarms or sending alerts for theft prevention [prohibited behaviors or actions]...”)
Claim 17: (Original)
Costello/Chen/Fail teaches the limitations upon which this claim depends. Furthermore, Costello in view of Fail teaches the following:
The method of claim 16, wherein receiving further includes pushing an up-to-date version of the menu to a mobile application of the mobile device using the mobile device identifier and receiving selections from the up-to-date version of the menu as the order based on customer interaction with the mobile application. (Fail, see at least [0025] Reference now being made to FIG. 3, a flow chart is shown illustrating the various communications between the communication system 200 and the dedicated device 100 of FIG. 1 according to the teachings of the present invention. The communication begins with initialization of the dedicated device 100 so that it is updated with the latest menu items (e.g. special of the day), any other information that is capable of being updated ( e.g. voice recognition software), and selection of the individuals preferred language (Step 302). The initialization can also include the assignment of one or more particular service individuals to be associated with the dedicated device (e.g. table assignments)…”)
Claim 19: (currently amended)
Pertaining to claim 1, as shown, Costello teaches the following:
A system, comprising:
cameras configured to capture time-stamped videos of patrons both outside and inside an
establishment (Costello, see at least Fig. 3, [0030], and [0064]-[0074]. E.g. per [0030]: “It is noted that the metadata collected for the processed images from the video captured during the customer's visit includes data and time stamps and camera identifiers for each of the cameras 120 and 130 that created, added to, or deleted from the metadata…”);
a server comprising a processor and a non-transitory computer-readable storage medium; the non-transitory computer-readable storage medium comprising executable instructions (Costello, see citations noted supra including at least [0064]: “…The real-time video tracker is implemented as executable instructions programmed and residing within memory and/or a non-transitory computer-readable (processor-readable) storage medium and executed by one or more processors of a hardware device…”);
the executable instructions when executed on the processor from the non-transitory computer-readable storage medium causing the processor to perform operations comprising:
authenticating a customer to a customer identity based on biometric features captured within the time-stamped videos (Costello, see citations noted supra, including again at least [0064]-[0074], e.g.: “…At 310, the real-time video tracker provides a unique identifier that is to be associated with a customer being tracked by decentralized video tracking agents within a store when a customer enters the store… the real-time video tracker identifies a customer identifier for the customer based on an automated check-in mechanism… (…automatically performed through facial recognition) when the customer enters the store… Then, the real-time video tracker accesses a customer database using the customer identifier (which was mapped by the real-time video tracker to the unique identifier and in an embodiment where the customer is registered with the store) to obtain a customer record that includes the customer's registered payment method for the transaction. Third-party payment services may then be used to complete and verify the payment processing for the transaction. In some cases, based on a customer profile associated with the customer record…”);
tracking locations of the customer within the establishment from the time-stamped
videos using video analytics that utilize […] and customer identifiers (Costello, see citations noted supra, e.g. [0068]-[0072]: “…At 310, the real-time video tracker provides a unique identifier [patron identifier] that is to be associated with a customer [patron] being tracked [tracked location] by decentralized video tracking agents within a store when a customer enters the store… The real-time video tracker links the customer identifier to the provided unique identifier… The identifiers allow the real-time video tracker to determine the physical locations of cameras for each of the tracking agents, the field-of-views for each tracking agent, and item attributes (including coordinates) for the real-time video tracker to identify items taken by the customer while in the store based on the unique identifier, the identifiers, and the metadata…”);
[…]
monitoring ordered items associated with the order […] using the video analytics that employ the bounding boxes and customer identifiers (Costello, see citations noted supra in view of at least Fig. 2, [0032]- [0035], and [047]-[0049], e.g.: per [0032]:”…The image tracking/transaction manager 141 is networked to the store's… item database (which includes each item's aisle location and shelf location within that aisle). The image tracking/transaction manager 141 processes the metadata… to search the item database for specific items…”; per [0057]: “…at 270, the next processing instance of the decentralized video agent modifies the metadata to include item attributes for an item detected as being picked up by the object within the next video frames being captured…”; per [0072]: “…The identifiers allow the real-time video tracker to determine the physical locations of cameras for each of the tracking agents, the field-of-views for each tracking agent, and item attributes (including coordinates) for the real-time video tracker to identify items taken by the customer while in the store based on the unique identifier, the identifiers, and the metadata….”);
[…]
automatically processing a payment for the order (Costello, see citations noted supra, e.g. per [0074]: “…at 331, the real-time video tracker reconstructs an entire transaction for the items taken and the customer and completes payment for the transaction based on a registered payment method associated with the customer. That is, once the items taken are identified, the real-time video tracker interacts with a store item database to obtain item descriptions and item pricing and generates a receipt and a total price for the transaction…”) when the customer is detected as exiting the establishment from the time-stamped videos video based on the corresponding […] customer identifier being tracked in the time-stamped videos (Costello, see citations noted supra in view of at least [0061]-[0062], teaching e.g.: “…At some point, the object (customer) is detected as approaching an egress point of the store [an indication, as determined by the video analytics, the certain patron exits the establishment]. For purposes of illustration this is identified by the further processing instance, in an embodiment of 271, at 272. The further processing instance creates final metadata from the modified metadata when the object (customer) is detected at or proximate to an exit point of the store within the further (third) video frames…”; the difference between the claims and the prior art of record is only that Costello appears to teach that his processing (i.e. of a payment for an item [a service item] taken by his customer, where the payment uses a registered payment method) occurs either when customer takes the item or approaches the egress/exit but not explicitly “when” the customer exits. However, examiner finds that there is a need to make a determination that a customer is exiting if a customer comes within some very close proximity of an egress and it is within the level of skill of a person of ordinary skill in the art to establish some proximity to an exit as an indication of a high-probability that a customer is existing or is in the process of exiting relative to the time at which the system may be processing such information and therefore, there is motivation to modify Costello’s teaching to process his payment transaction when the certain patron exits the establishment as determined by the video analytics because per MPEP 2143(I) (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference teachings to arrive at the claimed invention is obvious. The motivation may be implicit and may be found in the knowledge of one of ordinary skill in the art, or, in some cases, from the nature of the problem to be solved. Id. at 1366, 80 USPQ2d at 1649.)
Although Costello teaches the above limitations including real-time time-stamped video monitoring and tracking of the movement of customers and the items taken by such customer moving within a “frictionless store” environment where such customers do not need to directly interact with a service person to pay for items taken (e.g. including items being consumed), Costello may not explicitly contemplate the nuance recited below regarding his video and object tracking technology employing “bounding boxes” within the pixels of his time-stamped videos. However, regarding this nuanced feature, Costello in view of Chen teaches the following:
video analytics that employ bounding boxes; and …corresponding bounding boxes; […]
[monitoring…] by assigning an action identifier, a behavior identifier, or a gesture identifier to a certain bounding box linked to a corresponding customer identifier for the customer within the time-stamped videos (Chen, see at least Figs. 19, 25 and at least [0073]-[0078], teaching, e.g.: “…Video analytics also provides various other features. For example, video analytics can operate as an Intelligent Video Motion [action] Detector by detecting moving objects and by tracking moving objects. In some cases, the video analytics can generate and display a bounding box around a valid object… Video analytics can further be used to perform various types of recognition functions, such as face detection and recognition, …, or other recognition functions. In some cases, video analytics can be trained to recognize certain objects. Another function that can be performed by video analytics includes providing demographics for customer metrics (e.g., customer counts, gender, age, amount of time spent, and other suitable metrics)…”; other metrics suggests customer identification as well as per [0099] teaching: “…In some cases, a tracker is assigned with a unique ID, and a history of bounding boxes is kept. Object tracking in a video sequence can be used for many applications, including surveillance applications, among many others…” Also see at least [0007]: “…An object classified by the complex object detector can be localized using a bounding region (e.g., a bounding box or other bounding region) representing the classified object….”)
Therefore, the Examiner understands that the limitation in question is merely applying a known technique of Chen (directed towards video analytics and object recognition which makes use of bounding boxes around valid objects for the purpose of object classification) which is applicable to a known base device/method of Costello (already directed towards using video analytics to identify people and objects, e.g. via facial recognition and similar techniques) to yield predictable results. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the techniques of Chen to the device/method of Costello to improve Costello’s person and object recognition system and method because Chen is pertinent to the video analytics and object recognition aspects of Costello and because according to MPEP 2143(I) (C) and/or (D), the use of known technique to improve a known device, methods, or products in the same way (or which is ready for improvement) is obvious.
Although Costello/Chen teaches the above limitations including monitoring and tracking the movement of customers, as well as monitoring and identifying items taken by such customer moving within a “frictionless store” environment where such customers do not need to directly interact with a service person to pay for items taken (e.g. including items being consumed), and Costello teaches he alerts service staff of activities which may require their interaction with a customer, e.g. such as potential theft as described at least per [0092], Costello may not explicitly contemplate the nuanced situation and method of monitoring a service item being prepared for a customer’s request nor of alerting/dispatching service staff to deliver such items when ready for delivery. However, regarding these features, Costello in view of Fail teaches the following:
notifying a service staff member when at least one ordered item is completed based on the
videos and providing a current location of the customer to the staff member (Fail, see again at least [0037]-[0038], teaching e.g.: “…Once the food or other service item is ready [when at least one ordered item is completed] for delivery to the individual, the waiter or other responsible individual is prompted [notifying/dispatching a service staff member] via pager or other electronic device 210 to pick-up and deliver the food to the waiting individual [implies location is known/provided to staff member to enable said delivery]…”; Examiner notes that the state or indication of being “ready” is what is used to “notify” a service staff member. A fact that the state of “ready” was determined predicated upon other data, e.g. “based on the video” is non-functional as relates to this particular claimed step. Nonetheless, the prior art of Costello teaches such video analytics and it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have recognized the video analytics of Costello may be used to inform the decision of readiness which Fail uses to page his waiter or other individual to delivery food, i.e. the thing that is determined to be in a state of readiness, to a waiting individual. Furthermore, the location of such individual would be obvious to communicate to the waiter to enable the delivery because per MPEP 2143(I) (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention is obvious. The motivation to combine may be implicit and may be found in the knowledge of one of ordinary skill in the art, or, in some cases, from the nature of the problem to be solved. Id. at 1366, 80 USPQ2d at 1649.)
monitoring the customer from the time-stamped videos for an action, a behavior, or a
gesture indicating that a service request is needed by the customer [Fail, see at least [0007] teaching: “…places such as restaurants where menus are provided in a printed format. The individual will often require a specialized menu in a Braille format or assistance from another individual (e.g. waiter) to read the menu to them…” and per [0035]: “…It should also be noted that the individual can, at any time, page [monitoring the customer for an action indicating a service request is needed by the customer] a waiter for additional assistance…”; see [0033]-[0038])
[monitoring ordered items associated with the order] for completing by the service
preparation staff (Fail, see at least [0037]-[0038], teaching, e.g.: “…Once the food or other service item is ready for delivery to the individual, the waiter or other responsible individual is prompted via pager or other electronic device 210 to pick-up and deliver the food to the waiting individual…”; applicant’s “monitoring” reads on Fail’s determination that the service item is “ready”)
notifying the service preparation staff or the service staff member of the service request of the service request and the current location of the customer based on the service request; (Fail, see again at least [0037]-[0038], teaching e.g.: “…Once the food or other service item is ready for delivery to the individual, the waiter or other responsible individual is prompted via pager or other electronic device 210 to pick-up and deliver the food to the waiting individual…”; Examiner notes that a notification to a waiter, to deliver food to a waiting individual, within this context is also an implied indication that the waiting individual has made a service request for such food and the location of the waiting individual would be obvious to communicate to enable delivery of such food as already contemplated by Fail. Furthermore, Examiner notes, it would be obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to make a separate communication/notification to the waiter that the waiting individual had indeed made a service request to order the food which the waiter is also notified to deliver, e.g. to confirm to the waiter payment has been received, or to avoid confusion in delivering the food to the proper waiting individual, etc… and because per MPEP 2144.04 (V)(C) – Making separable is obvious.)
Therefore, the Examiner understands that the limitations in question are merely applying known techniques of Fail (directed towards techniques of monitoring for the readiness of an item being prepared for a customer at a customer’s request) which are applicable to a known base device/method of Costello (already directed towards monitoring service items as well as customers via cameras to identify which items a customer takes or receives in order to electronically bill the customer in a “frictionless store” environment) to yield predictable results. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the techniques of Fail to the device/method of Costello such that Costello also provides this service per the technique of Fail to his customers to enhance the services provided by his “frictionless store” to his customers because Fail is pertinent to the customer service aspects of Costello and because according to MPEP 2143(I) (C) and/or (D), the use of known technique to improve a known device, methods, or products in the same way (or which is ready for improvement) is obvious.
Claim 20: (previously presented)
Costello/Chen/Fail teaches the limitations upon which this claim depends. Furthermore, Costello teaches the following:
The system of claim 19, wherein the executable instructions when executed on the processor from the non-transitory computer-readable storage medium further cause the processor to perform additional operations comprising:
monitoring the service preparation staff, the service staff member, and the customer from the time-stamped videos for a prohibited action or a prohibited behavior using the video analytics; (Costello, see citations noted supra in view of at least [0033] -[0034], [0075], and [0092], e.g. per [0034]: “…It is to be noted that the above-noted processing can be used for security purposes as well to detect and thwart theft [prohibited action]…”; and per [0075]: “…The real-time video tracker sends an alert to staff devices for any of the items taken that are unaccounted for in the items that were scam1ed when the customer attempts to end the transaction and pay for the items that were scam1ed on the transaction terminal.”; Examiner notes that it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have extended Costello’s monitoring of his customers to monitor all persons [i.e. the service preparation staff, the service staff member, and the customer] for possible theft, by duplicating his existing techniques to include identifiers for his service staff and/or store personnel because per MPEP 2144.04 (VI)(B) – Duplication is obvious) sending a message to an external system or a designated resource based on a severity level assigned to the prohibited action or the prohibited behavior (Costello, see at least [0092]: “…Still further, the video transaction manager 402 can process the action as a mechanism for raising alarms or sending notifications to store equipment when theft [prohibited action] is suspected [a level of severity] by the customer based on the metadata. In an embodiment, the video transaction manager 402 can process multiple actions for a frictionless-store transaction, verifying a transaction terminal transaction, and raising automated alarms or sending alerts for theft prevention [prohibited behaviors or actions]...”)
Claims 6 are rejected under 35 U.S.C. 103 as obvious over Costello in view of Chen, Fail, and Arslan et al (U.S. 2015/0156328 A1; hereinafter, "Arslan").
Claim 6: (previously presented)
Although Costello/Chen/Fail teach the limitations upon which this claim depends, including customer verification, e.g. via biometrics such as facial recognition amongst various techniques, they may not explicitly teach the nuance as recited below directed towards authenticating a customer via voice sample. However, regarding this feature, Costello in view of Arslan teaches the following:
The method of claim 1, wherein authenticating further includes obtaining a voice sample from the certain patron at the location from a microphone and authenticating the certain patron to the identity based on the voice sample. (Arslan, see at least [0057], e.g.: “…Thus, the user ID will be approved upon the biometric identification thereof. Voice authentication can be made by asking the user to repeat a random sentence or by the answer thereof provided for the security question…”)
Therefore, the Examiner understands that the limitation in question is merely applying a known technique of Arslan which is applicable to a known base device/method of Costello to yield predictable results. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the techniques of Arslan to the device/method of Costello because Arslan is pertinent to the customer authentication objective of Costello and because according to MPEP 2143(I) (C) and/or (D), the use of known technique to improve a known device, methods, or products in the same way (or which is ready for improvement) is obvious.
Claims 8 are rejected under 35 U.S.C. 103 as obvious over Costello in view of Chen, Fail and Shanmugam et al (U.S. 2021/0110144 A1; hereinafter, "Shanmugam").
Claim 8: (previously presented)
Although Costello/Chen/Fail teach the limitations upon which this claim depends, and Costello/Fail teach indicating that the certain patron desires a refill of a particular beverage (e.g. see Fail Figs. 6-8 and associated disclosure, e.g. [0029]-[0034]; customer indicates they want a “drink” such as a “soda” from the menu) they may not explicitly teach receiving such request via an observed gesture as per the nuance recited below. However, regarding this feature, Costello/Fail in view of Shanmugam teaches the following:
The method of claim 7 further comprising, identifying a particular gesture made by the certain patron within the time-stamped videos […] (Shanmugam, see at least Figs. 3-4 and [0037]-[0045], e.g.: “…In step 302, the customer 104 may perform a physical gesture while located in the physical area 108. The physical gesture may be captured by one or more optical imaging devices 106… the determination module 222 of the processing server 102 may determine that the physical gesture matches a predefined gesture,… processing server 102 may electronically transmit a notification message to the employee 110 via a computing device 112 associated therewith. The notification message may include at least the identified customer status as well as the geographic location of the customer 104 and the predefined gesture that was performed by the customer 104… The employee 110 may then, in step 314, provide assistance to the customer 104 utilizing the information provided in the notification message regarding the customer's status and the assistance they requested based on the predefined gesture that was performed. In step 316, the customer 104 may receive the assistance provided by the employee 110 as a result of their performing the predefined gesture in the physical area 108…”)
Therefore, the Examiner understands that the limitation in question is merely applying a known technique of Shanmugam (directed towards specific techniques of using gesture recognition to receive indications from customers regarding requested services and/or service items) which is applicable to a known base device/method of Costello/Fail (already directed towards system/method of receiving customer indications of desired services and/or service items from a menu) to yield predictable results. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the techniques of Shanmugam to the device/method of Costello/Fail such that Costell/Fail identifying a particular gesture made by their customer(s) [a certain patron] within their time-stamped videos as the mechanism by which to receive their customer’s indication the customer wants a “drink” such as a “soda” from the menu [indicating that the certain patron desires a refill of a particular beverage] because Shanmugam is pertinent to the customer service objectives of Costello/Fail and because according to MPEP 2143(I) (C) and/or (D), the use of known technique to improve a known device, methods, or products in the same way (or which is ready for improvement) is obvious.
Claim 9: (previously presented)
Although Costello/Chen/Fail teach the limitations upon which this claim depends, and Costello/Fail teach indicating that the certain patron desires the service staff member to come to the location to assist the certain patron, (e.g. see Fail Figs. 6-8 and associated disclosure, e.g. [0007] “The
individual will often require a specialized menu in a Braille format or assistance from another individual (e.g. waiter) to read the menu to them.” and per [0035], teaching: “…It should also be noted that the individual can, at any time, page a waiter for additional assistance.”) they may not explicitly teach receiving such request via an observed gesture as per the nuance recited below. However, regarding this feature, Costello/Fail in view of Shanmugam and Official Notice teaches the following:
The method of claim 7 further comprising, identifying a particular gesture made by the certain patron within the time-stamped videos […]; and notifying the service staff member to go the location and assist the certain patron (Shanmugam, see at least Figs. 3-4 and [0037]-[0045], e.g.: “…In step 302, the customer 104 may perform a physical gesture while located in the physical area 108. The physical gesture may be captured by one or more optical imaging devices 106… the determination module 222 of the processing server 102 may determine that the physical gesture matches a predefined gesture,… processing server 102 may electronically transmit a notification message [notifying] to the employee 110 [the service staff member] via a computing device 112 associated therewith. The notification message may include at least the identified customer status as well as the geographic location of the customer 104 and the predefined gesture that was performed by the customer 104… The employee 110 may then, in step 314, provide assistance to the customer 104 utilizing the information provided in the notification message regarding the customer's status and the assistance they requested based on the predefined gesture that was performed. In step 316, the customer 104 may receive the assistance provided by the employee 110 as a result of their performing the predefined gesture in the physical area 108…”)
Therefore, the Examiner understands that the limitation in question is merely applying a known technique of Shanmugam (directed towards specific techniques of using gesture recognition to receive indications from customers regarding requested services such as assistance at their location) which is applicable to a known base device/method of Costello/Fail (already directed towards system/method of receiving customer indications of desired services such as assistance to help customer at their location read a menu) to yield predictable results. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the techniques of Shanmugam to the device/method of Costello/Fail such that Costell/Fail identifying a particular gesture made by the certain patron within the time-stamped videos as the mechanism by which to receive their customer’s indication the customer wants assistance (e.g. to read a menu) [indicating that the certain patron desires the service staff member to come to the location to assist the certain patron] and then notifying the service staff member to go the location and assist the certain patron because Shanmugam is pertinent to the customer service objectives of Costello/Fail and because according to MPEP 2143(I) (C) and/or (D), the use of known technique to improve a known device, methods, or products in the same way (or which is ready for improvement) is obvious.
Claim 13: (previously presented)
Although Costello/Chen/Fail teach the limitations upon which this claim depends, and Costello/Fail teach use of video analytics to identify customers and teach generating a service request to the service preparation staff or the service staff member based on an action, a behavior (e.g. see Fail Figs. 6-8 and associated disclosure, e.g. [0029]-[0034]; customer indicates they want a “drink” such as a “soda” from the menu) they may not explicitly teach detecting such via video analytics. However, regarding this feature, Costello/Fail in view of Shanmugam teaches the following:
The method of claim 12 further comprising, generating a service request to the service preparation staff or the service staff member based on an action, a behavior, or a gesture made by the customer and detected from the video analytics. (Shanmugam, see at least Figs. 3-4 and [0037]-[0045], e.g.: “…In step 302, the customer 104 may perform a physical gesture while located in the physical area 108. The physical gesture may be captured by one or more optical imaging devices 106… the determination module 222 of the processing server 102 may determine that the physical gesture matches a predefined gesture,… processing server 102 may electronically transmit a notification message [notifying] to the employee 110 [the service staff member] via a computing device 112 associated therewith. The notification message may include at least the identified customer status as well as the geographic location of the customer 104 and the predefined gesture that was performed by the customer 104… The employee 110 may then, in step 314, provide assistance to the customer 104 utilizing the information provided in the notification message regarding the customer's status and the assistance they requested based on the predefined gesture that was performed. In step 316, the customer 104 may receive the assistance provided by the employee 110 as a result of their performing the predefined gesture in the physical area 108…”)
Therefore, the Examiner understands that the limitation in question is merely applying a known technique of Shanmugam (directed towards specific techniques of using gesture recognition to receive indications from customers regarding requested services and/or service items) which is applicable to a known base device/method of Costello/Fail (already directed towards system/method of receiving customer indications of desired services and/or service items) to yield predictable results. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the techniques of Shanmugam to the device/method of Costello/Fail such that Costell/Fail also generate their service request to the service preparation staff or the service staff member based on an action, a behavior, or a gesture made by their customer and detected from their video analytics because Shanmugam is pertinent to the customer service objectives of Costello/Fail and because according to MPEP 2143(I) (C) and/or (D), the use of known technique to improve a known device, methods, or products in the same way (or which is ready for improvement) is obvious.
Claim 16 is rejected under 35 U.S.C. 103 as obvious over Costello in view of Chen, Fail further in view of Eby et al (US 11,157,929 B1; hereinafter, “Eby”).
Claim 16. (Original)
Although Costello/Chen/Fail teach the limitations upon which this claim depends, and Costello teaches authentication of a customer using his video analytics, he may not explicitly teach the below nuance. However, regarding the following features, Costello in view of Eby teaches the following: The method of claim 12, wherein authenticating further includes linking the customer identity to a registered customer profile comprising, a mobile device identifier for the mobile device, a loyalty account of the customer, a name of the customer, an image of the customer, and a registered payment method for processing payment of the order (Eby, see at least [] Customer database 120 may be configured to store any relevant customer data related to operation of the environment 100. Such data may include, for example, customer profile data that includes customer information (e.g., name; mailing address, email address, mobile phone number, home phone number, and/or other contact information; date of birth; age; gender; Social Security Number); user identifier 116 (e.g., a loyalty account number, a member identification (ID) number, patient ID number, a username, or other customer identifying information); a picture of the customer; account history information (e.g., online shopping history, in-store shopping history, shopping preferences, average purchase price in-store, average purchase price online, other spending habits, date last visited the retail store 200 or another enterprise location, or any other information related to previous account activity); a customer qualifier or label for categorizing the customer based on the account history of the customer and/or other profile data, the qualifier being selected from a list of pre-defined qualifiers, or customer segments, (e.g., “Medicare Part-D,” “Photo Aficionado,” “High-Value,” “High-Value Rx,” “Low-Value Repeat,” etc.); payment information (e.g., credit card information, debit card information, etc.); and any other profile information for each customer of the retail store 200.)
Therefore, the Examiner understands that the limitation in question is merely applying a known technique of Eby which is applicable to a known base device/method of Costello to yield predictable results. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to apply the techniques of Eby to the device/method of Costello in order to arrive at the limitation as claimed because Eby is pertinent to the authentication of Costello and because according to MPEP 2143(I) (C) and/or (D), the use of known technique to improve a known device, methods, or products in the same way (or which is ready for improvement) is obvious.
Claims 10-11, 18 are rejected under 35 U.S.C. 103 as obvious over Costello in view of Chen, Fail further in view of Official Notice.
Claim 10. (Original)
Although Costello/Chen/Fail teach the limitations upon which this claim depends, they may not explicitly teach the below nuance. However, regarding the following features, Costello/Chen/Fail in view of Official Notice teaches the following: The method of claim 1, wherein dispatching further includes identifying the service item being placed in a service staff pickup location by a service staff preparation member based on the monitoring and sending a message to a service staff operated device instructing the service staff member to take the service item to the location (Examiner takes Official Notice of the following facts: restaurants such as family diners have long operated to manually perform the identifying the service item, e.g. identifying by a waitress or waiter where cooks place food, which has been prepared per customer order, in an area designated as ready for the wait service staff to pick-up the prepared food and then deliver the food to the customer according to a ticket designating the customer and/or customer table which ordered the food. Additionally, a cook often will message the service staff, e.g. a waiter or waitress, using a bell or audio signal such as a whistle.) In view of these findings, the Examiner understands there is an existing need and motivation to automate such manual restaurant operations using techniques and technology (e.g. cameras, electronic devices, video analytics systems which use bounding boxes associated with image classifiers, etc…) as is known in the art before the effective filing date of the claimed invention, e.g. such as disclosed by Costello/Chen/Fail. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have performed the method steps as now claimed by making use of the technology already disclosed by Costello/Chen/Fail to identifying the service item being placed in a service staff pickup location by a service staff preparation member based on the monitoring and sending a message to a service staff operated device instructing the service staff member to take the service item to the location because per MPEP 2144.04 (III) – Automating a manual activity is obvious and according to MPEP 2143(I) (C) and/or (D) the use of known technique to improve a known device, methods, or products in the same way (or which is ready for improvement) is obvious.
Claim 11. (Original)
Although Costello/Chen/Fail teach the limitations upon which this claim depends, they may not explicitly teach the below nuance. However, regarding the following features, Costello/Chen/Fail in view of Official Notice teaches the following: The method of claim 1, wherein dispatching further includes updating the location as the certain patron moves about the establishment, wherein the location associated with the certain patron when the service request was made is different from the location associated with certain patron when the service staff member is dispatched to deliver the service item to the certain patron. (Examiner takes Official Notice of the following facts: restaurants such as family diners have long operated to manually perform the updating the location as the certain patron moves about the establishment, e.g. by visually following a patron who moves from bar seat to bar seat or to a different table which is more suitable for his party) In view of these findings, the Examiner understands there is an existing need and motivation to automate such manual restaurant operations using techniques and technology (e.g. cameras, electronic devices, video analytics systems which use bounding boxes associated with image classifiers, etc…) as is known in the art before the effective filing date of the claimed invention, e.g. such as disclosed by Costello/Chen/Fail. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have performed the method steps as now claimed by making use of the technology already disclosed by Costello/Chen/Fail to updating the location as the certain patron moves about the establishment, wherein the location associated with the certain patron when the service request was made is different from the location associated with certain patron when the service staff member is dispatched to deliver the service item to the certain patron because per MPEP 2144.04 (III) – Automating a manual activity is obvious and according to MPEP 2143(I) (C) and/or (D) the use of known technique to improve a known device, methods, or products in the same way (or which is ready for improvement) is obvious.
Claim 18. (Original)
Although Costello/Chen/Fail teach the limitations upon which this claim depends, they may not explicitly teach the below nuance. However, regarding the following features, Costello/Chen/Fail in view of Official Notice teaches the following: The method of claim 17, wherein notifying further includes sending a message to a service staff member device operated by the service staff member indicating a pickup location of order items for the order, the name of the customer, the image of the customer, and the current location of the customer within the establishment. (Examiner takes Official Notice of the following facts: restaurants such as family diners have long operated to manually perform the sending a message to a service staff member indicating a pickup location of order items for the order, the name of the customer, the image of the customer, and the current location of the customer within the establishment, e.g. by a manager watching the dinning hall and managing the service staff to describe a customer to a staff member with detail such that food may be properly picked up and delivered to the correct customer ordering the food) In view of these findings, the Examiner understands there is an existing need and motivation to automate such manual restaurant operations using techniques and technology (e.g. cameras, electronic devices, video analytics systems which use bounding boxes associated with image classifiers, etc…) as is known in the art before the effective filing date of the claimed invention, e.g. such as disclosed by Costello/Chen/Fail. Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have performed the method steps as now claimed by making use of the technology already disclosed by Costello/Chen/Fail to updating the location as the certain patron moves about the establishment, wherein the location associated with the certain patron when the service request was made is different from the location associated with certain patron when the service staff member is dispatched to deliver the service item to the certain patron because per MPEP 2144.04 (III) – Automating a manual activity is obvious and according to MPEP 2143(I) (C) and/or (D) the use of known technique to improve a known device, methods, or products in the same way (or which is ready for improvement) is obvious.
Response to Arguments
Applicant amended claims 1, 12, 19 on 06/26/2025. Applicant's arguments (hereinafter “Remarks”) also filed 05/23/2024, have been fully considered but are moot in view of the new grounds of rejection necessitated by applicant’s amendments. Note the new 35 USC 103 rejections with Costello in view of Chen and Fail. Also note the previous 35 USC 101 rejection has been withdrawn in view of consideration of applicant’s amendments and step 2B of the 2019 PEG.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL J SITTNER whose telephone number is (571)270-3984. The examiner can normally be reached M-F; ~9:30-6:30. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Waseem Ashraf can be reached on (571) 270-3948. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Michael J Sittner/
Primary Examiner, Art Unit 3621