DETAILED ACTION
Response to Arguments
The arguments filed 6/26/2026 have been entered and made of record.
The Applicant's arguments filed 6/26/2026 have been considered but they are not persuasive:
Re claim 1, Applicant states that the cited references WEXLER as modified by BUIBAS do not disclose the limitation “determine a target processor from the image signal processor and the event processor based on the motion distance, wherein a processing capability of the target processor matches the motion distance”;
However, the Examiner disagrees, because:
WEXLER as modified by BUIBAS disclose “determine a target processor from the image signal processor and the event processor based on the motion distance, wherein a processing capability of the target processor matches the motion distance” (see BUIBAS: e.g., --The processor receives a respective time sequence of images from cameras in the store, wherein the time sequence of images is captured over a time period and analyzes the time sequence of images from each camera and the 3D model of the store to detect a person in the store based on the time sequence of images, calculate a trajectory of the person across the time period, identify an item storage area of the item storage areas that is proximal to the trajectory of the person during an interaction time period within the time period, analyze two or more images of the time sequence of images to identify an item of the items within the item storage area that moves during the interaction time period, wherein the two or more images are captured within or proximal in time to the interaction time period and the two or more images contain views of the item storage area and attribute motion of the item to the person.--, in [0016]; and, --the processor is further configured to analyze the two or more images of the time sequence of images to classify the motion of the item as a type of motion comprising taking, putting or moving. [0022] In one or more embodiments, the processor analyzes two or more images of the time sequence of images to identify an item within the item storage area that moves during the interaction time period. Specifically, the processor uses or obtains a neural network trained to recognize items from changes across images, sets an input layer of the neural network to the two or more images and calculates a probability associated with the item based on an output layer of the neural network. In one or more embodiments, the neural network is further trained to classify an action performed on an item into classes comprising taking, putting, or moving. In one or more embodiments, the system includes a verification system configured to accept input confirming or denying that the person is associated with motion of the item. In one or more embodiments, the system includes a machine learning system configured to receive the input confirming or denying that the person is associated with the motion of the item and updates the neural network based on the input. Embodiments of the invention may utilize a neural network or more generally, any type of generic function approximator. By definition the function to map inputs of before-after image pairs, or before-during-after image pairs to output actions, then the neural network can be trained to be any such function map--, in [0021]-[0022], and, --0037] In one or more embodiments, tracking of the person may also occur in the secured environment, using cameras in the secured environment. As described above with respect to an automated store, tracking may determine when the person is near an item storage area, and analysis of two or more images of the item storage area may determine that an item has moved. Combining these analyses allows the system to attribute motion of an item to the person, and to charge the item to the person's account if the authorization is linked to a payment account. Again as described with respect to an automated store, tracking and determining when a person is at or near an item storage area may include calculating a 3D field of influence volume around the person; determining when an item is moved or taken may use a neural network that inputs two or more images (such as before and after images) of the item storage area and outputs a probability that an item is moved.--, in [0037]-[0038]; and, --the weight change location may be calculated as the weighted average of the weight sensor locations, with the weights equal to the weight changes associated with each weight sensor. [0082] In one or more embodiments the processor may calculate a change location confidence based on the distance between the weight change location and the visual change region of interest.--, in [0081]-[0082], and, --[0223] Various methods may be used to calculate a 3D field of influence volume around a person. FIGS. 6A through 6E illustrate a method that may be used in one or more embodiments. (These figures illustrate the construction of a field of influence volume using 2D figures, for ease of illustration, but the method may be applied in three dimensions to build a 3D volume around the person.) Based on an image or images 601 of a person, image analysis may be used to identify landmarks on the person's body. For example, landmark 602 may be the left elbow of the person. FIG. 6B illustrates an analysis process that identifies 18 different landmarks on the person's body. One or more embodiments may identify any number of landmarks on a body, at any desired level of detail. Landmarks may be connected in a skeleton in order to track the movement of the person's joints. Once landmark locations are identified in the 3D space associated with the store, one method for constructing a 3D field of influence volume is to calculate a sphere around each landmark with a radius of a specified threshold distance--, in [0223], --[0274] To track moving objects, in particular people, one or more embodiments of the system may incorporate a background subtraction or motion filter algorithm, masking out the background from the foreground for each of the planar projected images.--, in [0274]; and, --[0286] Appearance extraction from image 30G may for example be done by histograms, or by any other dimensionality reduction method. A lower dimensional vector may be formed from the composite image of each tracked person and used to compare it with other tracked subjects. For example, a neural network may be trained to take composite cylindrical images as input, and to output a lower-dimensional vector that is close to other vectors from the same person and far from vectors from other persons. To distinguish between people, vector-to-vector distances may be computed and compared to a threshold; for example, a distance of 0.0 to 0.5 may indicate the same person, and a greater distance may indicate different people. One or more embodiments may compare tracks of people by forming distributions of appearance vectors for each track, and comparing distributions using a distribution-to-distribution measure (such as KL-divergence, for example). A discriminant between distributions may be computed to label a new vector to an existing person in a store or site.--, in [0286], and, --[0368] During store operation, the quantity sensors may feed data into the signal processor 6610 which collects statistics on quantity measurements such as distance, weight, or other variables, and reports as a data packet of amount changed (distance/weight/other quantity variables) and time of start and end of the change.--, in [0368]),
It is clear that first, above BUIBAS discloses 1) image signal processor, which “analyzes the time sequence of images from each camera and the 3D model of the store to detect a person in the store based on the time sequence of images, calculate a trajectory of the person across the time period, identify an item storage area of the item storage areas that is proximal to the trajectory of the person during an interaction time period within the time period, analyze two or more images of the time sequence of images to identify an item of the items within the item storage area that moves during the interaction time period”, as recited in [0016]; and
an 2) event processor, which --the processor is further configured to analyze the two or more images of the time sequence of images to classify the motion of the item as a type of motion comprising taking, putting or moving.--, in [0021];
and, discloses “determining a target processor” such as in: --[0022] In one or more embodiments, the processor analyzes two or more images of the time sequence of images to identify an item within the item storage area that moves during the interaction time period. Specifically, the processor uses or obtains a neural network trained to recognize items from changes across images, sets an input layer of the neural network to the two or more images and calculates a probability associated with the item based on an output layer of the neural network. In one or more embodiments, the neural network is further trained to classify an action performed on an item into classes comprising taking, putting, or moving. In one or more embodiments, the system includes a verification system configured to accept input confirming or denying that the person is associated with motion of the item. In one or more embodiments, the system includes a machine learning system configured to receive the input confirming or denying that the person is associated with the motion of the item and updates the neural network based on the input. Embodiments of the invention may utilize a neural network or more generally, any type of generic function approximator. By definition the function to map inputs of before-after image pairs, or before-during-after image pairs to output actions, then the neural network can be trained to be any such function map--, in [0022];
so that, the processor of [0022], which read on “determined a target processor” from 1) the image signal processor, of [0016], for “detecting a person”, and 2) the event processor, of [0021], for “classifying the motion of the item”, which is aligned to “event”, classifying/detecting/identifying and event;
and the processor of [0022], which read on “determined a target processor”, based on the “motion distance, wherein a processing capability of the target processor matches the motion distance” such as based on “to receive the input confirming or denying that the person is associated with the motion of the item and updates the neural network based on the input”, furthermore, the processor of [0022], which is further implemented as “trained neural network” to further tracking object/person, that “to classify an action performed on an item into classes comprising taking, putting, or moving.”; and herein BUIBAS’s “a lower-dimensional vector that is close to other vectors from the same person and far from vectors from other persons. To distinguish between people, vector-to-vector distances may be computed and compared to a threshold; for example, a distance of 0.0 to 0.5 may indicate the same person, and a greater distance may indicate different people”, read on the claimed element of “motion distance”, which match to a processing capability of the target processor, such as of “to identify an item within the item storage area that moves during the interaction time period.”, and/ or “to classify an action performed on an item into classes comprising taking, putting, or moving”, as disclosed in BUIBAS’s [0022], of “determining a target processor”;
Furthermore, above discussions BUIBAS’s disclosures of “determining a target processor”, in [0022], from 1) an image signal processor, in [0016], and 2) an event processor, in [0021], are consistent with WEXLER’s relevant disclosures of 1) an image signal processor, and 2) an event processor, as discussed in the Office Action, WEXLER also discloses the target processor to track the target object in images/ adjacent frames. Thus, WEXLER as modified by BUIBAS teach and disclose every limitations, and overall as whole of claim 1, and claims 1-20.
Therefore, claims 1-20 are still not patentably distinguishable over the prior art reference(s). Further discussions are addressed in the prior art rejection section below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over WEXLER (WO 2022130011 A1, also as the same published as US 20230336694 A1, claims the priority of parent US continuation PCT/IB2021/000834 20211130, and US-Provisional-Application US 63125537 20201215), and in view of BUIBAS (US 20210272086 A1).
Re Claim 1, WEXLER discloses an apparatus for tracking objects (see WEXLER : e.g., --the at least one tracking subsystem includes at least one processor programmed to: receive a plurality of images from the one or more cameras; identify at least one individual represented by the plurality of images; determine at least one characteristic of the at least one individual; and generate and send an alert based on the at least one characteristic.--, in abstract; and, -- tracking a movement of at least a portion of an object included in one or more images captured by the image sensor. For example, in one embodiment, apparatus 110 may track an object as long as the object remains substantially within the field-of-view of image sensor 220… monitoring an object or person captured by an image sensor 220 may include tracking movement of the object across the fields of view of the plurality of image sensors 220..--, in [0208]-[0209], and, -- the system may track statistical information associated with interactions with individuals. For example, the system may track interactions with each encountered individual and automatically update a personal record of interactions with the encountered individual. The system may provide analytics and tags per individual based on meeting context (e.g., work meeting, sports meeting, etc.). Information, such as a summary of the relationship, may be provided to the user via an interface. In some embodiments, the interface may order individuals chronologically based on analytics or tags. For example, the system may group or order individuals by attendees at recent meetings, meeting location, amount of time spent together, or various other characteristics. Accordingly, the disclosed embodiments may provide, among other advantages, improved efficiency, convenience, and functionality over prior art wearable apparatuses.--, in [0237]-[0238], and, -- processor 210 may compare one or more images of the captured plurality of images (or characteristics such as color, pattern, shapes, etc. in the captured images) to a database of images/characteristics stored in memory 550a of apparatus 110 (and/or memory 550b of computing device 120) to identify that the user is drinking coffee… processor 210 may detect the type of beverage that user 3610 is drinking and/or track the quantity of the beverage consumed by user 3610.--, in [0509], [0524]-[0525]), comprising:
an image signal processor and an event processor, wherein a processing capability of the image signal processor is different from a processing capability of the event processor (see WEXLER : e.g., – At least one processor may be programmed to execute a method comprising analyzing at least one image from among the plurality of images to identify an event in which the user is involved--, in [0020]
-- [0509] Processor 210 may analyze the images captured by image sensor 220 to identify the activity that the user is engaged in and/or an action of the user (i.e., user action) during the activity (step 3830). That is, based on an analysis of the captured images while user 3610 is engaged with friends 3620, 3630, processor 210 may identify or recognize that the user action is drinking coffee.--, in [0509]-[0510] {herein read on claimed limitation of “an event processor}; and,
a system may include a user device comprising a camera configured to capture a plurality of images from an environment of a user and output an image signal comprising the plurality of images; and at least one processor. The at least one processor may be programmed to detect, in at least one of the plurality of images, a face of an individual represented in the at least one of the plurality of images; isolate at least one facial feature of the detected face--, in [0024] {herein read on claimed limitation of “an image signal processor});
an image sensor configured to capture a first raw image; and a controller (see WEXLER : e.g., – At least one processor may be programmed to execute a method comprising analyzing at least one image from among the plurality of images to identify an event in which the user is involved--, in [0020]
-- [0509] Processor 210 may analyze the images captured by image sensor 220 to identify the activity that the user is engaged in and/or an action of the user (i.e., user action) during the activity (step 3830). That is, based on an analysis of the captured images while user 3610 is engaged with friends 3620, 3630, processor 210 may identify or recognize that the user action is drinking coffee.--, in [0509]-[0510] {herein read on claimed limitation of “an event processor}; and,
a system may include a user device comprising a camera configured to capture a plurality of images from an environment of a user and output an image signal comprising the plurality of images; and at least one processor. The at least one processor may be programmed to detect, in at least one of the plurality of images, a face of an individual represented in the at least one of the plurality of images; isolate at least one facial feature of the detected face--, in [0024]) configured to:
although WEXLER discloses determine motion characteristics and movement of the tracked objects (see WEXLER : e.g., -- monitoring module 603 may be provided for continuous monitoring. Such continuous monitoring may include tracking a movement of at least a portion of an object included in one or more images captured by the image sensor…. monitoring an object or person captured by an image sensor 220 may include tracking movement of the object across the fields of view of the plurality of image sensors 220.--, in [0208]-[0209], and, -- the individual may be recognized based on other physical characteristics or traits. For example, the system may detect a body shape or posture of the individual, which may indicate an identity of the individual. Similarly, an individual may have particular gestures, mannerisms (e.g., movement of hands, facial movements, gait, typing or writing patterns, eye movements, or other bodily movements) that the system may use to identify the individual.--, in [0239]; and, -- he at least one processor may be programmed to execute a method comprising receiving a first motion signal indicative of an output of at least one of a first motion sensor or a first location sensor of a mobile device. For example, motion sensor (e.g., accelerometer 6950) of mobile device 120 may sense a motion or change in velocity or acceleration of mobile device 120. For example, user 100 may be carrying or wearing mobile device 120 (e.g., a smartphone) and may be walking, running, riding, and/or traveling in, for example, a land-based, sea-based, or airborne vehicle. Accelerometer 6950 may periodically or continuously generate signals representative of the detected motion or change in velocity or acceleration of mobile device 120.--, in [0838]-[0839], and, -- determining whether the mobile device and the wearable device share the one or more motion characteristics includes determining whether the first motion signal and the second motion signal differ relative to one or more thresholds. For example, it is contemplated that processor 210 may determine one or more differences between various parameters (e.g., positions, speeds, accelerations, velocities, directions of movement, etc., over one or more periods of time) associated with wearable device 110 and mobile device 120. It is contemplated that differences may be obtained in many ways, for example, vector distance, cosine distance, or by performing other mathematical operations known in the art for determining differences. By way of example, processor 210 may determine differences between the positions of wearable device 110 and mobile device 120 over a plurality of time periods. Furthermore, processor 210 may compare the determined differences with one or more thresholds. Processor 210 may determine that wearable device 110 and mobile device 120 share one or more motion characteristics when the corresponding differences are about zero, or are less than corresponding correlation thresholds.--, in [0851]);
WEXLER however does not explicitly disclose determine a motion distance in two adjacent frames of raw images of a target object in the first raw image received from the image sensor, wherein the first raw image is a next frame of an image of the two adjacent frames of raw images;
BUIBAS discloses determine a motion distance in two adjacent frames of raw images of a target object in the first raw image received from the image sensor, wherein the first raw image is a next frame of an image of the two adjacent frames of raw images (see BUIBAS: e.g., --The processor receives a respective time sequence of images from cameras in the store, wherein the time sequence of images is captured over a time period and analyzes the time sequence of images from each camera and the 3D model of the store to detect a person in the store based on the time sequence of images, calculate a trajectory of the person across the time period, identify an item storage area of the item storage areas that is proximal to the trajectory of the person during an interaction time period within the time period, analyze two or more images of the time sequence of images to identify an item of the items within the item storage area that moves during the interaction time period, wherein the two or more images are captured within or proximal in time to the interaction time period and the two or more images contain views of the item storage area and attribute motion of the item to the person.--, in [0016]; and, --he processor is further configured to analyze the two or more images of the time sequence of images to classify the motion of the item as a type of motion comprising taking, putting or moving.
[0022] In one or more embodiments, the processor analyzes two or more images of the time sequence of images to identify an item within the item storage area that moves during the interaction time period. Specifically, the processor uses or obtains a neural network trained to recognize items from changes across images, sets an input layer of the neural network to the two or more images and calculates a probability associated with the item based on an output layer of the neural network. In one or more embodiments, the neural network is further trained to classify an action performed on an item into classes comprising taking, putting, or moving. In one or more embodiments, the system includes a verification system configured to accept input confirming or denying that the person is associated with motion of the item. In one or more embodiments, the system includes a machine learning system configured to receive the input confirming or denying that the person is associated with the motion of the item and updates the neural network based on the input. Embodiments of the invention may utilize a neural network or more generally, any type of generic function approximator. By definition the function to map inputs of before-after image pairs, or before-during-after image pairs to output actions, then the neural network can be trained to be any such function map--, in [0021], and, --0037] In one or more embodiments, tracking of the person may also occur in the secured environment, using cameras in the secured environment. As described above with respect to an automated store, tracking may determine when the person is near an item storage area, and analysis of two or more images of the item storage area may determine that an item has moved. Combining these analyses allows the system to attribute motion of an item to the person, and to charge the item to the person's account if the authorization is linked to a payment account. Again as described with respect to an automated store, tracking and determining when a person is at or near an item storage area may include calculating a 3D field of influence volume around the person; determining when an item is moved or taken may use a neural network that inputs two or more images (such as before and after images) of the item storage area and outputs a probability that an item is moved.--, in [0037]-[0038]; and, --the weight change location may be calculated as the weighted average of the weight sensor locations, with the weights equal to the weight changes associated with each weight sensor. [0082] In one or more embodiments the processor may calculate a change location confidence based on the distance between the weight change location and the visual change region of interest.--, in [0081]-[0082], and, --[0223] Various methods may be used to calculate a 3D field of influence volume around a person. FIGS. 6A through 6E illustrate a method that may be used in one or more embodiments. (These figures illustrate the construction of a field of influence volume using 2D figures, for ease of illustration, but the method may be applied in three dimensions to build a 3D volume around the person.) Based on an image or images 601 of a person, image analysis may be used to identify landmarks on the person's body. For example, landmark 602 may be the left elbow of the person. FIG. 6B illustrates an analysis process that identifies 18 different landmarks on the person's body. One or more embodiments may identify any number of landmarks on a body, at any desired level of detail. Landmarks may be connected in a skeleton in order to track the movement of the person's joints. Once landmark locations are identified in the 3D space associated with the store, one method for constructing a 3D field of influence volume is to calculate a sphere around each landmark with a radius of a specified threshold distance--, in [0223], --[0274] To track moving objects, in particular people, one or more embodiments of the system may incorporate a background subtraction or motion filter algorithm, masking out the background from the foreground for each of the planar projected images.--, in [0274]; and, --[0286] Appearance extraction from image 30G may for example be done by histograms, or by any other dimensionality reduction method. A lower dimensional vector may be formed from the composite image of each tracked person and used to compare it with other tracked subjects. For example, a neural network may be trained to take composite cylindrical images as input, and to output a lower-dimensional vector that is close to other vectors from the same person and far from vectors from other persons. To distinguish between people, vector-to-vector distances may be computed and compared to a threshold; for example, a distance of 0.0 to 0.5 may indicate the same person, and a greater distance may indicate different people. One or more embodiments may compare tracks of people by forming distributions of appearance vectors for each track, and comparing distributions using a distribution-to-distribution measure (such as KL-divergence, for example). A discriminant between distributions may be computed to label a new vector to an existing person in a store or site.--, in [0286], and, --[0368] During store operation, the quantity sensors may feed data into the signal processor 6610 which collects statistics on quantity measurements such as distance, weight, or other variables, and reports as a data packet of amount changed (distance/weight/other quantity variables) and time of start and end of the change.--, in [0368]);
WEXLER and BUIBAS are combinable as they are in the same field of endeavor: tracking objects associated with events. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify WEXLER ’s apparatus using BUIBAS’s teachings by including determine a motion distance in two adjacent frames of raw images of a target object in the first raw image received from the image sensor, wherein the first raw image is a next frame of an image of the two adjacent frames of raw images to WEXLER ’s detecting the movement of objects, such as change locations in order to (see BUIBAS: e.g., in [0016], [0021]-[0022] [0037]-[0038], [0081]-[0082], [0223], [0274], [0268], and [0386]);
WEXLER as modified by BUIBAS further disclose determine a target processor from the image signal processor and the event processor based on the motion distance, wherein a processing capability of the target processor matches the motion distance (see BUIBAS: e.g., --The processor receives a respective time sequence of images from cameras in the store, wherein the time sequence of images is captured over a time period and analyzes the time sequence of images from each camera and the 3D model of the store to detect a person in the store based on the time sequence of images, calculate a trajectory of the person across the time period, identify an item storage area of the item storage areas that is proximal to the trajectory of the person during an interaction time period within the time period, analyze two or more images of the time sequence of images to identify an item of the items within the item storage area that moves during the interaction time period, wherein the two or more images are captured within or proximal in time to the interaction time period and the two or more images contain views of the item storage area and attribute motion of the item to the person.--, in [0016]; and, --he processor is further configured to analyze the two or more images of the time sequence of images to classify the motion of the item as a type of motion comprising taking, putting or moving. [0022] In one or more embodiments, the processor analyzes two or more images of the time sequence of images to identify an item within the item storage area that moves during the interaction time period. Specifically, the processor uses or obtains a neural network trained to recognize items from changes across images, sets an input layer of the neural network to the two or more images and calculates a probability associated with the item based on an output layer of the neural network. In one or more embodiments, the neural network is further trained to classify an action performed on an item into classes comprising taking, putting, or moving. In one or more embodiments, the system includes a verification system configured to accept input confirming or denying that the person is associated with motion of the item. In one or more embodiments, the system includes a machine learning system configured to receive the input confirming or denying that the person is associated with the motion of the item and updates the neural network based on the input. Embodiments of the invention may utilize a neural network or more generally, any type of generic function approximator. By definition the function to map inputs of before-after image pairs, or before-during-after image pairs to output actions, then the neural network can be trained to be any such function map--, in [0021], and, --0037] In one or more embodiments, tracking of the person may also occur in the secured environment, using cameras in the secured environment. As described above with respect to an automated store, tracking may determine when the person is near an item storage area, and analysis of two or more images of the item storage area may determine that an item has moved. Combining these analyses allows the system to attribute motion of an item to the person, and to charge the item to the person's account if the authorization is linked to a payment account. Again as described with respect to an automated store, tracking and determining when a person is at or near an item storage area may include calculating a 3D field of influence volume around the person; determining when an item is moved or taken may use a neural network that inputs two or more images (such as before and after images) of the item storage area and outputs a probability that an item is moved.--, in [0037]-[0038]; and, --the weight change location may be calculated as the weighted average of the weight sensor locations, with the weights equal to the weight changes associated with each weight sensor. [0082] In one or more embodiments the processor may calculate a change location confidence based on the distance between the weight change location and the visual change region of interest.--, in [0081]-[0082], and, --[0223] Various methods may be used to calculate a 3D field of influence volume around a person. FIGS. 6A through 6E illustrate a method that may be used in one or more embodiments. (These figures illustrate the construction of a field of influence volume using 2D figures, for ease of illustration, but the method may be applied in three dimensions to build a 3D volume around the person.) Based on an image or images 601 of a person, image analysis may be used to identify landmarks on the person's body. For example, landmark 602 may be the left elbow of the person. FIG. 6B illustrates an analysis process that identifies 18 different landmarks on the person's body. One or more embodiments may identify any number of landmarks on a body, at any desired level of detail. Landmarks may be connected in a skeleton in order to track the movement of the person's joints. Once landmark locations are identified in the 3D space associated with the store, one method for constructing a 3D field of influence volume is to calculate a sphere around each landmark with a radius of a specified threshold distance--, in [0223], --[0274] To track moving objects, in particular people, one or more embodiments of the system may incorporate a background subtraction or motion filter algorithm, masking out the background from the foreground for each of the planar projected images.--, in [0274]; and, --[0286] Appearance extraction from image 30G may for example be done by histograms, or by any other dimensionality reduction method. A lower dimensional vector may be formed from the composite image of each tracked person and used to compare it with other tracked subjects. For example, a neural network may be trained to take composite cylindrical images as input, and to output a lower-dimensional vector that is close to other vectors from the same person and far from vectors from other persons. To distinguish between people, vector-to-vector distances may be computed and compared to a threshold; for example, a distance of 0.0 to 0.5 may indicate the same person, and a greater distance may indicate different people. One or more embodiments may compare tracks of people by forming distributions of appearance vectors for each track, and comparing distributions using a distribution-to-distribution measure (such as KL-divergence, for example). A discriminant between distributions may be computed to label a new vector to an existing person in a store or site.--, in [0286], and, --[0368] During store operation, the quantity sensors may feed data into the signal processor 6610 which collects statistics on quantity measurements such as distance, weight, or other variables, and reports as a data packet of amount changed (distance/weight/other quantity variables) and time of start and end of the change.--, in [0368]), and
perform target tracking on the target object based on the first raw image using the target processor ((see BUIBAS: e.g., --The processor receives a respective time sequence of images from cameras in the store, wherein the time sequence of images is captured over a time period and analyzes the time sequence of images from each camera and the 3D model of the store to detect a person in the store based on the time sequence of images, calculate a trajectory of the person across the time period, identify an item storage area of the item storage areas that is proximal to the trajectory of the person during an interaction time period within the time period, analyze two or more images of the time sequence of images to identify an item of the items within the item storage area that moves during the interaction time period, wherein the two or more images are captured within or proximal in time to the interaction time period and the two or more images contain views of the item storage area and attribute motion of the item to the person.--, in [0016]; and, --he processor is further configured to analyze the two or more images of the time sequence of images to classify the motion of the item as a type of motion comprising taking, putting or moving. [0022] In one or more embodiments, the processor analyzes two or more images of the time sequence of images to identify an item within the item storage area that moves during the interaction time period. Specifically, the processor uses or obtains a neural network trained to recognize items from changes across images, sets an input layer of the neural network to the two or more images and calculates a probability associated with the item based on an output layer of the neural network. In one or more embodiments, the neural network is further trained to classify an action performed on an item into classes comprising taking, putting, or moving. In one or more embodiments, the system includes a verification system configured to accept input confirming or denying that the person is associated with motion of the item. In one or more embodiments, the system includes a machine learning system configured to receive the input confirming or denying that the person is associated with the motion of the item and updates the neural network based on the input. Embodiments of the invention may utilize a neural network or more generally, any type of generic function approximator. By definition the function to map inputs of before-after image pairs, or before-during-after image pairs to output actions, then the neural network can be trained to be any such function map--, in [0021], and, --0037] In one or more embodiments, tracking of the person may also occur in the secured environment, using cameras in the secured environment. As described above with respect to an automated store, tracking may determine when the person is near an item storage area, and analysis of two or more images of the item storage area may determine that an item has moved. Combining these analyses allows the system to attribute motion of an item to the person, and to charge the item to the person's account if the authorization is linked to a payment account. Again as described with respect to an automated store, tracking and determining when a person is at or near an item storage area may include calculating a 3D field of influence volume around the person; determining when an item is moved or taken may use a neural network that inputs two or more images (such as before and after images) of the item storage area and outputs a probability that an item is moved.--, in [0037]-[0038]; and, --the weight change location may be calculated as the weighted average of the weight sensor locations, with the weights equal to the weight changes associated with each weight sensor. [0082] In one or more embodiments the processor may calculate a change location confidence based on the distance between the weight change location and the visual change region of interest.--, in [0081]-[0082], and, --[0223] Various methods may be used to calculate a 3D field of influence volume around a person. FIGS. 6A through 6E illustrate a method that may be used in one or more embodiments. (These figures illustrate the construction of a field of influence volume using 2D figures, for ease of illustration, but the method may be applied in three dimensions to build a 3D volume around the person.) Based on an image or images 601 of a person, image analysis may be used to identify landmarks on the person's body. For example, landmark 602 may be the left elbow of the person. FIG. 6B illustrates an analysis process that identifies 18 different landmarks on the person's body. One or more embodiments may identify any number of landmarks on a body, at any desired level of detail. Landmarks may be connected in a skeleton in order to track the movement of the person's joints. Once landmark locations are identified in the 3D space associated with the store, one method for constructing a 3D field of influence volume is to calculate a sphere around each landmark with a radius of a specified threshold distance--, in [0223], --[0274] To track moving objects, in particular people, one or more embodiments of the system may incorporate a background subtraction or motion filter algorithm, masking out the background from the foreground for each of the planar projected images.--, in [0274]; and, --[0286] Appearance extraction from image 30G may for example be done by histograms, or by any other dimensionality reduction method. A lower dimensional vector may be formed from the composite image of each tracked person and used to compare it with other tracked subjects. For example, a neural network may be trained to take composite cylindrical images as input, and to output a lower-dimensional vector that is close to other vectors from the same person and far from vectors from other persons. To distinguish between people, vector-to-vector distances may be computed and compared to a threshold; for example, a distance of 0.0 to 0.5 may indicate the same person, and a greater distance may indicate different people. One or more embodiments may compare tracks of people by forming distributions of appearance vectors for each track, and comparing distributions using a distribution-to-distribution measure (such as KL-divergence, for example). A discriminant between distributions may be computed to label a new vector to an existing person in a store or site.--, in [0286], and, --[0368] During store operation, the quantity sensors may feed data into the signal processor 6610 which collects statistics on quantity measurements such as distance, weight, or other variables, and reports as a data packet of amount changed (distance/weight/other quantity variables) and time of start and end of the change.--, in [0368]; also see BUIBAS: e.g., --The processor receives a respective time sequence of images from cameras in the store, wherein the time sequence of images is captured over a time period and analyzes the time sequence of images from each camera and the 3D model of the store to detect a person in the store based on the time sequence of images, calculate a trajectory of the person across the time period, identify an item storage area of the item storage areas that is proximal to the trajectory of the person during an interaction time period within the time period, analyze two or more images of the time sequence of images to identify an item of the items within the item storage area that moves during the interaction time period, wherein the two or more images are captured within or proximal in time to the interaction time period and the two or more images contain views of the item storage area and attribute motion of the item to the person.--, in [0016]; and, --he processor is further configured to analyze the two or more images of the time sequence of images to classify the motion of the item as a type of motion comprising taking, putting or moving. [0022] In one or more embodiments, the processor analyzes two or more images of the time sequence of images to identify an item within the item storage area that moves during the interaction time period. Specifically, the processor uses or obtains a neural network trained to recognize items from changes across images, sets an input layer of the neural network to the two or more images and calculates a probability associated with the item based on an output layer of the neural network. In one or more embodiments, the neural network is further trained to classify an action performed on an item into classes comprising taking, putting, or moving. In one or more embodiments, the system includes a verification system configured to accept input confirming or denying that the person is associated with motion of the item. In one or more embodiments, the system includes a machine learning system configured to receive the input confirming or denying that the person is associated with the motion of the item and updates the neural network based on the input. Embodiments of the invention may utilize a neural network or more generally, any type of generic function approximator. By definition the function to map inputs of before-after image pairs, or before-during-after image pairs to output actions, then the neural network can be trained to be any such function map--, in [0021], and, --0037] In one or more embodiments, tracking of the person may also occur in the secured environment, using cameras in the secured environment. As described above with respect to an automated store, tracking may determine when the person is near an item storage area, and analysis of two or more images of the item storage area may determine that an item has moved. Combining these analyses allows the system to attribute motion of an item to the person, and to charge the item to the person's account if the authorization is linked to a payment account. Again as described with respect to an automated store, tracking and determining when a person is at or near an item storage area may include calculating a 3D field of influence volume around the person; determining when an item is moved or taken may use a neural network that inputs two or more images (such as before and after images) of the item storage area and outputs a probability that an item is moved.--, in [0037]-[0038]; and, --the weight change location may be calculated as the weighted average of the weight sensor locations, with the weights equal to the weight changes associated with each weight sensor. [0082] In one or more embodiments the processor may calculate a change location confidence based on the distance between the weight change location and the visual change region of interest.--, in [0081]-[0082], and, --[0223] Various methods may be used to calculate a 3D field of influence volume around a person. FIGS. 6A through 6E illustrate a method that may be used in one or more embodiments. (These figures illustrate the construction of a field of influence volume using 2D figures, for ease of illustration, but the method may be applied in three dimensions to build a 3D volume around the person.) Based on an image or images 601 of a person, image analysis may be used to identify landmarks on the person's body. For example, landmark 602 may be the left elbow of the person. FIG. 6B illustrates an analysis process that identifies 18 different landmarks on the person's body. One or more embodiments may identify any number of landmarks on a body, at any desired level of detail. Landmarks may be connected in a skeleton in order to track the movement of the person's joints. Once landmark locations are identified in the 3D space associated with the store, one method for constructing a 3D field of influence volume is to calculate a sphere around each landmark with a radius of a specified threshold distance--, in [0223], --[0274] To track moving objects, in particular people, one or more embodiments of the system may incorporate a background subtraction or motion filter algorithm, masking out the background from the foreground for each of the planar projected images.--, in [0274]; and, --[0286] Appearance extraction from image 30G may for example be done by histograms, or by any other dimensionality reduction method. A lower dimensional vector may be formed from the composite image of each tracked person and used to compare it with other tracked subjects. For example, a neural network may be trained to take composite cylindrical images as input, and to output a lower-dimensional vector that is close to other vectors from the same person and far from vectors from other persons. To distinguish between people, vector-to-vector distances may be computed and compared to a threshold; for example, a distance of 0.0 to 0.5 may indicate the same person, and a greater distance may indicate different people. One or more embodiments may compare tracks of people by forming distributions of appearance vectors for each track, and comparing distributions using a distribution-to-distribution measure (such as KL-divergence, for example). A discriminant between distributions may be computed to label a new vector to an existing person in a store or site.--, in [0286], and, --[0368] During store operation, the quantity sensors may feed data into the signal processor 6610 which collects statistics on quantity measurements such as distance, weight, or other variables, and reports as a data packet of amount changed (distance/weight/other quantity variables) and time of start and end of the change.--, in [0368]).
Re Claim 2, WEXLER as modified by BUIBAS further disclose if the target processor is the image signal processor, the target processor is configured to convert the first raw image to an RGB image and send the RGB image to the controller (see BUIBAS: e.g., --[0303] FIG. 39 illustrates how the position weight maps generated in FIG. 38 may be used in one or more embodiments for person detection. Projected images 3611 and 3612, from cameras 3411 and 3412, respectively, may be separated into color channels. FIG. 39 illustrates separating these images into RGB color channels; these channels are illustrative, and one or more embodiments may use any desired decomposition of images into channels using any color space or any other image processing methods. The RGB channels are combined with a fourth channel representing the position weight map for the camera that captured the image. The four channels for each image are input into machine learning system 3220, which generates an output 3221a with detection probabilities for each pixel. Therefore image 3611 corresponds to four inputs 3611r, 3611g, 3611b, and 3821; and image 3612 corresponds to four inputs 3612r, 3612g, 3612b, and 3822. To simplify the machine learning system, in one or more embodiments the position weight maps 3821 and 3822 may be scaled to have the same size as the associated color channels.--, in [0303]); or
if the target processor is the event processor, the target processor is configured to convert the first raw image to an event data stream and send the event data stream to the controller, wherein the event data stream indicates a brightness change status of a pixel in the first raw image; and the controller is configured to perform target tracking on the target object based on the RGB image or the event data stream (see WEXLER : e.g., – At least one processor may be programmed to execute a method comprising analyzing at least one image from among the plurality of images to identify an event in which the user is involved--, in [0020]; and, -- [0509] Processor 210 may analyze the images captured by image sensor 220 to identify the activity that the user is engaged in and/or an action of the user (i.e., user action) during the activity (step 3830). That is, based on an analysis of the captured images while user 3610 is engaged with friends 3620, 3630, processor 210 may identify or recognize that the user action is drinking coffee.--, in [0509]-[0510] {herein read on claimed limitation of “an event processor}; also see BUIBAS: e.g., --[0253] The trajectory 1920 calculated by processor 130, which may be updated as the person 1901 moves through the area, may associate locations with times. For example, person 1901 is at location 1921 at time 1922. In one or more embodiments the locations and the times may be ranges rather than specific points in space and time. These ranges may for example reflect uncertainties or limitations in measurement, or the effects of discrete sampling. For example, if a camera captures images every second, then a time associated with a location obtained from one camera image may be a time range with a width of two seconds. Sampling and extension of a trajectory with a new point may also occur in response to an event, such as a person entering a zone or triggering a sensor, instead of or in addition to sampling at a fixed frequency. Ranges for location may also reflect that a person occupies a volume in space, rather than a single point. This volume may for example be or be related to the 3D field of influence volume described above with respect to FIGS. 6A through 7B.--, [0253]).
Re Claim 3, WEXLER as modified by BUIBAS further disclose wherein the controller is configured to determine the target processor from the image signal processor and the event processor based on the motion distance and a preset distance threshold (see WEXLER : e.g., -- monitoring module 603 may be provided for continuous monitoring. Such continuous monitoring may include tracking a movement of at least a portion of an object included in one or more images captured by the image sensor…. monitoring an object or person captured by an image sensor 220 may include tracking movement of the object across the fields of view of the plurality of image sensors 220.--, in [0208]-[0209], and, -- the individual may be recognized based on other physical characteristics or traits. For example, the system may detect a body shape or posture of the individual, which may indicate an identity of the individual. Similarly, an individual may have particular gestures, mannerisms (e.g., movement of hands, facial movements, gait, typing or writing patterns, eye movements, or other bodily movements) that the system may use to identify the individual.--, in [0239]; and, -- he at least one processor may be programmed to execute a method comprising receiving a first motion signal indicative of an output of at least one of a first motion sensor or a first location sensor of a mobile device. For example, motion sensor (e.g., accelerometer 6950) of mobile device 120 may sense a motion or change in velocity or acceleration of mobile device 120. For example, user 100 may be carrying or wearing mobile device 120 (e.g., a smartphone) and may be walking, running, riding, and/or traveling in, for example, a land-based, sea-based, or airborne vehicle. Accelerometer 6950 may periodically or continuously generate signals representative of the detected motion or change in velocity or acceleration of mobile device 120.--, in [0838]-[0839], and, -- determining whether the mobile device and the wearable device share the one or more motion characteristics includes determining whether the first motion signal and the second motion signal differ relative to one or more thresholds. For example, it is contemplated that processor 210 may determine one or more differences between various parameters (e.g., positions, speeds, accelerations, velocities, directions of movement, etc., over one or more periods of time) associated with wearable device 110 and mobile device 120. It is contemplated that differences may be obtained in many ways, for example, vector distance, cosine distance, or by performing other mathematical operations known in the art for determining differences. By way of example, processor 210 may determine differences between the positions of wearable device 110 and mobile device 120 over a plurality of time periods. Furthermore, processor 210 may compare the determined differences with one or more thresholds. Processor 210 may determine that wearable device 110 and mobile device 120 share one or more motion characteristics when the corresponding differences are about zero, or are less than corresponding correlation thresholds.--, in [0851]; also see BUIBAS: e.g., --[0223] Various methods may be used to calculate a 3D field of influence volume around a person. FIGS. 6A through 6E illustrate a method that may be used in one or more embodiments. (These figures illustrate the construction of a field of influence volume using 2D figures, for ease of illustration, but the method may be applied in three dimensions to build a 3D volume around the person.) Based on an image or images 601 of a person, image analysis may be used to identify landmarks on the person's body. For example, landmark 602 may be the left elbow of the person. FIG. 6B illustrates an analysis process that identifies 18 different landmarks on the person's body. One or more embodiments may identify any number of landmarks on a body, at any desired level of detail. Landmarks may be connected in a skeleton in order to track the movement of the person's joints. Once landmark locations are identified in the 3D space associated with the store, one method for constructing a 3D field of influence volume is to calculate a sphere around each landmark with a radius of a specified threshold distance--, in [0223], --[0274] To track moving objects, in particular people, one or more embodiments of the system may incorporate a background subtraction or motion filter algorithm, masking out the background from the foreground for each of the planar projected images.--, in [0274]; and, --[0286] Appearance extraction from image 30G may for example be done by histograms, or by any other dimensionality reduction method. A lower dimensional vector may be formed from the composite image of each tracked person and used to compare it with other tracked subjects. For example, a neural network may be trained to take composite cylindrical images as input, and to output a lower-dimensional vector that is close to other vectors from the same person and far from vectors from other persons. To distinguish between people, vector-to-vector distances may be computed and compared to a threshold; for example, a distance of 0.0 to 0.5 may indicate the same person, and a greater distance may indicate different people. One or more embodiments may compare tracks of people by forming distributions of appearance vectors for each track, and comparing distributions using a distribution-to-distribution measure (such as KL-divergence, for example). A discriminant between distributions may be computed to label a new vector to an existing person in a store or site.--, in [0286], and, --[0368] During store operation, the quantity sensors may feed data into the signal processor 6610 which collects statistics on quantity measurements such as distance, weight, or other variables, and reports as a data packet of amount changed (distance/weight/other quantity variables) and time of start and end of the change.--, in [0368]).
Re Claim 4, WEXLER as modified by BUIBAS further disclose wherein the processing capability of the image signal processor is higher than the processing capability of the event processor, and the controller is configured to:
if the motion distance is greater than the preset distance threshold, determine the image signal processor as the target processor (see WEXLER : e.g., -- monitoring module 603 may be provided for continuous monitoring. Such continuous monitoring may include tracking a movement of at least a portion of an object included in one or more images captured by the image sensor…. monitoring an object or person captured by an image sensor 220 may include tracking movement of the object across the fields of view of the plurality of image sensors 220.--, in [0208]-[0209], and, -- the individual may be recognized based on other physical characteristics or traits. For example, the system may detect a body shape or posture of the individual, which may indicate an identity of the individual. Similarly, an individual may have particular gestures, mannerisms (e.g., movement of hands, facial movements, gait, typing or writing patterns, eye movements, or other bodily movements) that the system may use to identify the individual.--, in [0239]; and, -- he at least one processor may be programmed to execute a method comprising receiving a first motion signal indicative of an output of at least one of a first motion sensor or a first location sensor of a mobile device. For example, motion sensor (e.g., accelerometer 6950) of mobile device 120 may sense a motion or change in velocity or acceleration of mobile device 120. For example, user 100 may be carrying or wearing mobile device 120 (e.g., a smartphone) and may be walking, running, riding, and/or traveling in, for example, a land-based, sea-based, or airborne vehicle. Accelerometer 6950 may periodically or continuously generate signals representative of the detected motion or change in velocity or acceleration of mobile device 120.--, in [0838]-[0839], and, -- determining whether the mobile device and the wearable device share the one or more motion characteristics includes determining whether the first motion signal and the second motion signal differ relative to one or more thresholds. For example, it is contemplated that processor 210 may determine one or more differences between various parameters (e.g., positions, speeds, accelerations, velocities, directions of movement, etc., over one or more periods of time) associated with wearable device 110 and mobile device 120. It is contemplated that differences may be obtained in many ways, for example, vector distance, cosine distance, or by performing other mathematical operations known in the art for determining differences. By way of example, processor 210 may determine differences between the positions of wearable device 110 and mobile device 120 over a plurality of time periods. Furthermore, processor 210 may compare the determined differences with one or more thresholds. Processor 210 may determine that wearable device 110 and mobile device 120 share one or more motion characteristics when the corresponding differences are about zero, or are less than corresponding correlation thresholds.--, in [0851]; also see BUIBAS: e.g., --[0223] Various methods may be used to calculate a 3D field of influence volume around a person. FIGS. 6A through 6E illustrate a method that may be used in one or more embodiments. (These figures illustrate the construction of a field of influence volume using 2D figures, for ease of illustration, but the method may be applied in three dimensions to build a 3D volume around the person.) Based on an image or images 601 of a person, image analysis may be used to identify landmarks on the person's body. For example, landmark 602 may be the left elbow of the person. FIG. 6B illustrates an analysis process that identifies 18 different landmarks on the person's body. One or more embodiments may identify any number of landmarks on a body, at any desired level of detail. Landmarks may be connected in a skeleton in order to track the movement of the person's joints. Once landmark locations are identified in the 3D space associated with the store, one method for constructing a 3D field of influence volume is to calculate a sphere around each landmark with a radius of a specified threshold distance--, in [0223], --[0274] To track moving objects, in particular people, one or more embodiments of the system may incorporate a background subtraction or motion filter algorithm, masking out the background from the foreground for each of the planar projected images.--, in [0274]; and, --[0286] Appearance extraction from image 30G may for example be done by histograms, or by any other dimensionality reduction method. A lower dimensional vector may be formed from the composite image of each tracked person and used to compare it with other tracked subjects. For example, a neural network may be trained to take composite cylindrical images as input, and to output a lower-dimensional vector that is close to other vectors from the same person and far from vectors from other persons. To distinguish between people, vector-to-vector distances may be computed and compared to a threshold; for example, a distance of 0.0 to 0.5 may indicate the same person, and a greater distance may indicate different people. One or more embodiments may compare tracks of people by forming distributions of appearance vectors for each track, and comparing distributions using a distribution-to-distribution measure (such as KL-divergence, for example). A discriminant between distributions may be computed to label a new vector to an existing person in a store or site.--, in [0286], and, --[0368] During store operation, the quantity sensors may feed data into the signal processor 6610 which collects statistics on quantity measurements such as distance, weight, or other variables, and reports as a data packet of amount changed (distance/weight/other quantity variables) and time of start and end of the change.--, in [0368]); or
if the motion distance is less than or equal to the preset distance threshold, determine the event processor as the target processor (see WEXLER : e.g., -- monitoring module 603 may be provided for continuous monitoring. Such continuous monitoring may include tracking a movement of at least a portion of an object included in one or more images captured by the image sensor…. monitoring an object or person captured by an image sensor 220 may include tracking movement of the object across the fields of view of the plurality of image sensors 220.--, in [0208]-[0209], and, -- the individual may be recognized based on other physical characteristics or traits. For example, the system may detect a body shape or posture of the individual, which may indicate an identity of the individual. Similarly, an individual may have particular gestures, mannerisms (e.g., movement of hands, facial movements, gait, typing or writing patterns, eye movements, or other bodily movements) that the system may use to identify the individual.--, in [0239]; and, -- the at least one processor may be programmed to execute a method comprising receiving a first motion signal indicative of an output of at least one of a first motion sensor or a first location sensor of a mobile device. For example, motion sensor (e.g., accelerometer 6950) of mobile device 120 may sense a motion or change in velocity or acceleration of mobile device 120. For example, user 100 may be carrying or wearing mobile device 120 (e.g., a smartphone) and may be walking, running, riding, and/or traveling in, for example, a land-based, sea-based, or airborne vehicle. Accelerometer 6950 may periodically or continuously generate signals representative of the detected motion or change in velocity or acceleration of mobile device 120.--, in [0838]-[0839], and, -- determining whether the mobile device and the wearable device share the one or more motion characteristics includes determining whether the first motion signal and the second motion signal differ relative to one or more thresholds. For example, it is contemplated that processor 210 may determine one or more differences between various parameters (e.g., positions, speeds, accelerations, velocities, directions of movement, etc., over one or more periods of time) associated with wearable device 110 and mobile device 120. It is contemplated that differences may be obtained in many ways, for example, vector distance, cosine distance, or by performing other mathematical operations known in the art for determining differences. By way of example, processor 210 may determine differences between the positions of wearable device 110 and mobile device 120 over a plurality of time periods. Furthermore, processor 210 may compare the determined differences with one or more thresholds. Processor 210 may determine that wearable device 110 and mobile device 120 share one or more motion characteristics when the corresponding differences are about zero, or are less than corresponding correlation thresholds.--, in [0851]; also see BUIBAS: e.g., --[0223] Various methods may be used to calculate a 3D field of influence volume around a person. FIGS. 6A through 6E illustrate a method that may be used in one or more embodiments. (These figures illustrate the construction of a field of influence volume using 2D figures, for ease of illustration, but the method may be applied in three dimensions to build a 3D volume around the person.) Based on an image or images 601 of a person, image analysis may be used to identify landmarks on the person's body. For example, landmark 602 may be the left elbow of the person. FIG. 6B illustrates an analysis process that identifies 18 different landmarks on the person's body. One or more embodiments may identify any number of landmarks on a body, at any desired level of detail. Landmarks may be connected in a skeleton in order to track the movement of the person's joints. Once landmark locations are identified in the 3D space associated with the store, one method for constructing a 3D field of influence volume is to calculate a sphere around each landmark with a radius of a specified threshold distance--, in [0223], --[0274] To track moving objects, in particular people, one or more embodiments of the system may incorporate a background subtraction or motion filter algorithm, masking out the background from the foreground for each of the planar projected images.--, in [0274]; and, --[0286] Appearance extraction from image 30G may for example be done by histograms, or by any other dimensionality reduction method. A lower dimensional vector may be formed from the composite image of each tracked person and used to compare it with other tracked subjects. For example, a neural network may be trained to take composite cylindrical images as input, and to output a lower-dimensional vector that is close to other vectors from the same person and far from vectors from other persons. To distinguish between people, vector-to-vector distances may be computed and compared to a threshold; for example, a distance of 0.0 to 0.5 may indicate the same person, and a greater distance may indicate different people. One or more embodiments may compare tracks of people by forming distributions of appearance vectors for each track, and comparing distributions using a distribution-to-distribution measure (such as KL-divergence, for example). A discriminant between distributions may be computed to label a new vector to an existing person in a store or site.--, in [0286], and, --[0368] During store operation, the quantity sensors may feed data into the signal processor 6610 which collects statistics on quantity measurements such as distance, weight, or other variables, and reports as a data packet of amount changed (distance/weight/other quantity variables) and time of start and end of the change.--, in [0368]).
Re Claim 5, WEXLER as modified by BUIBAS further disclose wherein the controller is configured to determine first location information based on the first raw image using the target processor, wherein the first location information indicates a location of the target object at a capturing moment of the first raw image (see WEXLER : e.g., -- monitoring module 603 may be provided for continuous monitoring. Such continuous monitoring may include tracking a movement of at least a portion of an object included in one or more images captured by the image sensor…. monitoring an object or person captured by an image sensor 220 may include tracking movement of the object across the fields of view of the plurality of image sensors 220.--, in [0208]-[0209], and, -- the individual may be recognized based on other physical characteristics or traits. For example, the system may detect a body shape or posture of the individual, which may indicate an identity of the individual. Similarly, an individual may have particular gestures, mannerisms (e.g., movement of hands, facial movements, gait, typing or writing patterns, eye movements, or other bodily movements) that the system may use to identify the individual.--, in [0239]; and, -- the at least one processor may be programmed to execute a method comprising receiving a first motion signal indicative of an output of at least one of a first motion sensor or a first location sensor of a mobile device. For example, motion sensor (e.g., accelerometer 6950) of mobile device 120 may sense a motion or change in velocity or acceleration of mobile device 120. For example, user 100 may be carrying or wearing mobile device 120 (e.g., a smartphone) and may be walking, running, riding, and/or traveling in, for example, a land-based, sea-based, or airborne vehicle. Accelerometer 6950 may periodically or continuously generate signals representative of the detected motion or change in velocity or acceleration of mobile device 120.--, in [0838]-[0839], and, -- determining whether the mobile device and the wearable device share the one or more motion characteristics includes determining whether the first motion signal and the second motion signal differ relative to one or more thresholds. For example, it is contemplated that processor 210 may determine one or more differences between various parameters (e.g., positions, speeds, accelerations, velocities, directions of movement, etc., over one or more periods of time) associated with wearable device 110 and mobile device 120. It is contemplated that differences may be obtained in many ways, for example, vector distance, cosine distance, or by performing other mathematical operations known in the art for determining differences. By way of example, processor 210 may determine differences between the positions of wearable device 110 and mobile device 120 over a plurality of time periods. Furthermore, processor 210 may compare the determined differences with one or more thresholds. Processor 210 may determine that wearable device 110 and mobile device 120 share one or more motion characteristics when the corresponding differences are about zero, or are less than corresponding correlation thresholds.--, in [0851]; also see BUIBAS: e.g., --[0223] Various methods may be used to calculate a 3D field of influence volume around a person. FIGS. 6A through 6E illustrate a method that may be used in one or more embodiments. (These figures illustrate the construction of a field of influence volume using 2D figures, for ease of illustration, but the method may be applied in three dimensions to build a 3D volume around the person.) Based on an image or images 601 of a person, image analysis may be used to identify landmarks on the person's body. For example, landmark 602 may be the left elbow of the person. FIG. 6B illustrates an analysis process that identifies 18 different landmarks on the person's body. One or more embodiments may identify any number of landmarks on a body, at any desired level of detail. Landmarks may be connected in a skeleton in order to track the movement of the person's joints. Once landmark locations are identified in the 3D space associated with the store, one method for constructing a 3D field of influence volume is to calculate a sphere around each landmark with a radius of a specified threshold distance--, in [0223], --[0254] The processor 130 tracks person 1901 to location 1923 at time 1924, where credential reader 1905 is located. In one or more embodiments location 1923 may be the same as location 1921 where tracking begins; however, in one or more embodiments the person may be tracked in an area upon entering the area and may provide a credential at another time, such as upon entering or exiting a store. In one or more embodiments, multiple credential readers may be present; for example, the gas station in FIG. 19 may have several pay-at-the-pump stations at which customers can enter credentials. Using analysis of camera images, processor 130 may determine which credential reader a person uses to enter a credential, which allows the processor to associate an authorization with the person, as described below.
[0255] As a result of entering credential 1904 into credential reader 1905, an authorization 1907 is provided to gas pump 1902. This authorization, or related data, may also be transmitted to processor 130. The authorization may for example be sent as a message 1910 from the pump or credential reader, or directly from bank or payment processor (or another authorization service) 212. Processor 130 may associate this authorization with person 1901 by determining that the trajectory 1920 of the person is at or near the location of the credential reader 1904 at or near the time that the authorization message is received or the time that the credential is presented to the credential reader 1905. In embodiments with multiple credential readers in an area, the processor 130 may associate a particular authorization with a particular person by determining which credential reader that authorization is associated with and by correlating the time of that authorization and the location of that credential reader with the trajectories of one or more people to determine which person is at or near that credential reader at that time. In some situations, the person 1901 may wait at the credential reader 1905 until the authorization is received; therefore processor 130 may use either the time that the credential is presented or the time that the authorization is received to determine which person is associated with the authorization.
[0256] By determining that person 1901 is at or near location 1923 at or near time 1924, determining that location 1923 is the location of credential reader 1905 (or within a zone near the credential reader) and determining that authorization 1910 is associated with credential reader 1905 and is received at or near time 1924 (or is associated with presentation of a credential at or near time 1924), processor 130 may associate the authorization with the trajectory 1920 of person 1901 after time 1924. This association 1932 may for example add an extended tag 1933 to the trajectory that includes authorization information and may include account or credential information associated with the authorization. Processor 130 may also associate certain allowed actions with the authorization; these allowed actions may be specific to the application and may also be specific to the particular authorization obtained for each person or each credential.
[0257] Processor 130 then continues to track the trajectory 1920 of person 1901 to the location 1925 at time 1926.--, in [0254]-[0257]; and, --[0274] To track moving objects, in particular people, one or more embodiments of the system may incorporate a background subtraction or motion filter algorithm, masking out the background from the foreground for each of the planar projected images.--, in [0274]; and, --[0286] Appearance extraction from image 30G may for example be done by histograms, or by any other dimensionality reduction method. A lower dimensional vector may be formed from the composite image of each tracked person and used to compare it with other tracked subjects. For example, a neural network may be trained to take composite cylindrical images as input, and to output a lower-dimensional vector that is close to other vectors from the same person and far from vectors from other persons. To distinguish between people, vector-to-vector distances may be computed and compared to a threshold; for example, a distance of 0.0 to 0.5 may indicate the same person, and a greater distance may indicate different people. One or more embodiments may compare tracks of people by forming distributions of appearance vectors for each track, and comparing distributions using a distribution-to-distribution measure (such as KL-divergence, for example). A discriminant between distributions may be computed to label a new vector to an existing person in a store or site.--, in [0286], and, --[0368] During store operation, the quantity sensors may feed data into the signal processor 6610 which collects statistics on quantity measurements such as distance, weight, or other variables, and reports as a data packet of amount changed (distance/weight/other quantity variables) and time of start and end of the change.--, in [0368]).
Re Claim 6, WEXLER as modified by BUIBAS further disclose wherein the controller is configured to: determine second location information based on a second raw image, wherein the second raw image is a former frame of a raw image in the two adjacent frames of raw images, and the second location information indicates a location of the target object at a capturing moment of the second raw image (see WEXLER : e.g., -- monitoring module 603 may be provided for continuous monitoring. Such continuous monitoring may include tracking a movement of at least a portion of an object included in one or more images captured by the image sensor…. monitoring an object or person captured by an image sensor 220 may include tracking movement of the object across the fields of view of the plurality of image sensors 220.--, in [0208]-[0209], and, -- the individual may be recognized based on other physical characteristics or traits. For example, the system may detect a body shape or posture of the individual, which may indicate an identity of the individual. Similarly, an individual may have particular gestures, mannerisms (e.g., movement of hands, facial movements, gait, typing or writing patterns, eye movements, or other bodily movements) that the system may use to identify the individual.--, in [0239]; and, -- the at least one processor may be programmed to execute a method comprising receiving a first motion signal indicative of an output of at least one of a first motion sensor or a first location sensor of a mobile device. For example, motion sensor (e.g., accelerometer 6950) of mobile device 120 may sense a motion or change in velocity or acceleration of mobile device 120. For example, user 100 may be carrying or wearing mobile device 120 (e.g., a smartphone) and may be walking, running, riding, and/or traveling in, for example, a land-based, sea-based, or airborne vehicle. Accelerometer 6950 may periodically or continuously generate signals representative of the detected motion or change in velocity or acceleration of mobile device 120.--, in [0838]-[0839], and, -- determining whether the mobile device and the wearable device share the one or more motion characteristics includes determining whether the first motion signal and the second motion signal differ relative to one or more thresholds. For example, it is contemplated that processor 210 may determine one or more differences between various parameters (e.g., positions, speeds, accelerations, velocities, directions of movement, etc., over one or more periods of time) associated with wearable device 110 and mobile device 120. It is contemplated that differences may be obtained in many ways, for example, vector distance, cosine distance, or by performing other mathematical operations known in the art for determining differences. By way of example, processor 210 may determine differences between the positions of wearable device 110 and mobile device 120 over a plurality of time periods. Furthermore, processor 210 may compare the determined differences with one or more thresholds. Processor 210 may determine that wearable device 110 and mobile device 120 share one or more motion characteristics when the corresponding differences are about zero, or are less than corresponding correlation thresholds.--, in [0851]; also see BUIBAS: e.g., --[0223] Various methods may be used to calculate a 3D field of influence volume around a person. FIGS. 6A through 6E illustrate a method that may be used in one or more embodiments. (These figures illustrate the construction of a field of influence volume using 2D figures, for ease of illustration, but the method may be applied in three dimensions to build a 3D volume around the person.) Based on an image or images 601 of a person, image analysis may be used to identify landmarks on the person's body. For example, landmark 602 may be the left elbow of the person. FIG. 6B illustrates an analysis process that identifies 18 different landmarks on the person's body. One or more embodiments may identify any number of landmarks on a body, at any desired level of detail. Landmarks may be connected in a skeleton in order to track the movement of the person's joints. Once landmark locations are identified in the 3D space associated with the store, one method for constructing a 3D field of influence volume is to calculate a sphere around each landmark with a radius of a specified threshold distance--, in [0223], --[0254] The processor 130 tracks person 1901 to location 1923 at time 1924, where credential reader 1905 is located. In one or more embodiments location 1923 may be the same as location 1921 where tracking begins; however, in one or more embodiments the person may be tracked in an area upon entering the area and may provide a credential at another time, such as upon entering or exiting a store. In one or more embodiments, multiple credential readers may be present; for example, the gas station in FIG. 19 may have several pay-at-the-pump stations at which customers can enter credentials. Using analysis of camera images, processor 130 may determine which credential reader a person uses to enter a credential, which allows the processor to associate an authorization with the person, as described below.
[0255] As a result of entering credential 1904 into credential reader 1905, an authorization 1907 is provided to gas pump 1902. This authorization, or related data, may also be transmitted to processor 130. The authorization may for example be sent as a message 1910 from the pump or credential reader, or directly from bank or payment processor (or another authorization service) 212. Processor 130 may associate this authorization with person 1901 by determining that the trajectory 1920 of the person is at or near the location of the credential reader 1904 at or near the time that the authorization message is received or the time that the credential is presented to the credential reader 1905. In embodiments with multiple credential readers in an area, the processor 130 may associate a particular authorization with a particular person by determining which credential reader that authorization is associated with and by correlating the time of that authorization and the location of that credential reader with the trajectories of one or more people to determine which person is at or near that credential reader at that time. In some situations, the person 1901 may wait at the credential reader 1905 until the authorization is received; therefore processor 130 may use either the time that the credential is presented or the time that the authorization is received to determine which person is associated with the authorization.
[0256] By determining that person 1901 is at or near location 1923 at or near time 1924, determining that location 1923 is the location of credential reader 1905 (or within a zone near the credential reader) and determining that authorization 1910 is associated with credential reader 1905 and is received at or near time 1924 (or is associated with presentation of a credential at or near time 1924), processor 130 may associate the authorization with the trajectory 1920 of person 1901 after time 1924. This association 1932 may for example add an extended tag 1933 to the trajectory that includes authorization information and may include account or credential information associated with the authorization. Processor 130 may also associate certain allowed actions with the authorization; these allowed actions may be specific to the application and may also be specific to the particular authorization obtained for each person or each credential.
[0257] Processor 130 then continues to track the trajectory 1920 of person 1901 to the location 1925 at time 1926.--, in [0254]-[0257]; and, --[0274] To track moving objects, in particular people, one or more embodiments of the system may incorporate a background subtraction or motion filter algorithm, masking out the background from the foreground for each of the planar projected images.--, in [0274]; and, --[0286] Appearance extraction from image 30G may for example be done by histograms, or by any other dimensionality reduction method. A lower dimensional vector may be formed from the composite image of each tracked person and used to compare it with other tracked subjects. For example, a neural network may be trained to take composite cylindrical images as input, and to output a lower-dimensional vector that is close to other vectors from the same person and far from vectors from other persons. To distinguish between people, vector-to-vector distances may be computed and compared to a threshold; for example, a distance of 0.0 to 0.5 may indicate the same person, and a greater distance may indicate different people. One or more embodiments may compare tracks of people by forming distributions of appearance vectors for each track, and comparing distributions using a distribution-to-distribution measure (such as KL-divergence, for example). A discriminant between distributions may be computed to label a new vector to an existing person in a store or site.--, in [0286], and, --[0368] During store operation, the quantity sensors may feed data into the signal processor 6610 which collects statistics on quantity measurements such as distance, weight, or other variables, and reports as a data packet of amount changed (distance/weight/other quantity variables) and time of start and end of the change.--, in [0368]);
determine third location information based on a third raw image, wherein the third raw image is a latter frame of the raw image in the two adjacent frames of raw images, and the third location information indicates a location of the target object at a capturing moment of the third raw image (see WEXLER : e.g., -- monitoring module 603 may be provided for continuous monitoring. Such continuous monitoring may include tracking a movement of at least a portion of an object included in one or more images captured by the image sensor…. monitoring an object or person captured by an image sensor 220 may include tracking movement of the object across the fields of view of the plurality of image sensors 220.--, in [0208]-[0209], and, -- the individual may be recognized based on other physical characteristics or traits. For example, the system may detect a body shape or posture of the individual, which may indicate an identity of the individual. Similarly, an individual may have particular gestures, mannerisms (e.g., movement of hands, facial movements, gait, typing or writing patterns, eye movements, or other bodily movements) that the system may use to identify the individual.--, in [0239]; and, -- the at least one processor may be programmed to execute a method comprising receiving a first motion signal indicative of an output of at least one of a first motion sensor or a first location sensor of a mobile device. For example, motion sensor (e.g., accelerometer 6950) of mobile device 120 may sense a motion or change in velocity or acceleration of mobile device 120. For example, user 100 may be carrying or wearing mobile device 120 (e.g., a smartphone) and may be walking, running, riding, and/or traveling in, for example, a land-based, sea-based, or airborne vehicle. Accelerometer 6950 may periodically or continuously generate signals representative of the detected motion or change in velocity or acceleration of mobile device 120.--, in [0838]-[0839], and, -- determining whether the mobile device and the wearable device share the one or more motion characteristics includes determining whether the first motion signal and the second motion signal differ relative to one or more thresholds. For example, it is contemplated that processor 210 may determine one or more differences between various parameters (e.g., positions, speeds, accelerations, velocities, directions of movement, etc., over one or more periods of time) associated with wearable device 110 and mobile device 120. It is contemplated that differences may be obtained in many ways, for example, vector distance, cosine distance, or by performing other mathematical operations known in the art for determining differences. By way of example, processor 210 may determine differences between the positions of wearable device 110 and mobile device 120 over a plurality of time periods. Furthermore, processor 210 may compare the determined differences with one or more thresholds. Processor 210 may determine that wearable device 110 and mobile device 120 share one or more motion characteristics when the corresponding differences are about zero, or are less than corresponding correlation thresholds.--, in [0851]; also see BUIBAS: e.g., --[0223] Various methods may be used to calculate a 3D field of influence volume around a person. FIGS. 6A through 6E illustrate a method that may be used in one or more embodiments. (These figures illustrate the construction of a field of influence volume using 2D figures, for ease of illustration, but the method may be applied in three dimensions to build a 3D volume around the person.) Based on an image or images 601 of a person, image analysis may be used to identify landmarks on the person's body. For example, landmark 602 may be the left elbow of the person. FIG. 6B illustrates an analysis process that identifies 18 different landmarks on the person's body. One or more embodiments may identify any number of landmarks on a body, at any desired level of detail. Landmarks may be connected in a skeleton in order to track the movement of the person's joints. Once landmark locations are identified in the 3D space associated with the store, one method for constructing a 3D field of influence volume is to calculate a sphere around each landmark with a radius of a specified threshold distance--, in [0223], --[0254] The processor 130 tracks person 1901 to location 1923 at time 1924, where credential reader 1905 is located. In one or more embodiments location 1923 may be the same as location 1921 where tracking begins; however, in one or more embodiments the person may be tracked in an area upon entering the area and may provide a credential at another time, such as upon entering or exiting a store. In one or more embodiments, multiple credential readers may be present; for example, the gas station in FIG. 19 may have several pay-at-the-pump stations at which customers can enter credentials. Using analysis of camera images, processor 130 may determine which credential reader a person uses to enter a credential, which allows the processor to associate an authorization with the person, as described below.
[0255] As a result of entering credential 1904 into credential reader 1905, an authorization 1907 is provided to gas pump 1902. This authorization, or related data, may also be transmitted to processor 130. The authorization may for example be sent as a message 1910 from the pump or credential reader, or directly from bank or payment processor (or another authorization service) 212. Processor 130 may associate this authorization with person 1901 by determining that the trajectory 1920 of the person is at or near the location of the credential reader 1904 at or near the time that the authorization message is received or the time that the credential is presented to the credential reader 1905. In embodiments with multiple credential readers in an area, the processor 130 may associate a particular authorization with a particular person by determining which credential reader that authorization is associated with and by correlating the time of that authorization and the location of that credential reader with the trajectories of one or more people to determine which person is at or near that credential reader at that time. In some situations, the person 1901 may wait at the credential reader 1905 until the authorization is received; therefore processor 130 may use either the time that the credential is presented or the time that the authorization is received to determine which person is associated with the authorization.
[0256] By determining that person 1901 is at or near location 1923 at or near time 1924, determining that location 1923 is the location of credential reader 1905 (or within a zone near the credential reader) and determining that authorization 1910 is associated with credential reader 1905 and is received at or near time 1924 (or is associated with presentation of a credential at or near time 1924), processor 130 may associate the authorization with the trajectory 1920 of person 1901 after time 1924. This association 1932 may for example add an extended tag 1933 to the trajectory that includes authorization information and may include account or credential information associated with the authorization. Processor 130 may also associate certain allowed actions with the authorization; these allowed actions may be specific to the application and may also be specific to the particular authorization obtained for each person or each credential.
[0257] Processor 130 then continues to track the trajectory 1920 of person 1901 to the location 1925 at time 1926.--, in [0254]-[0257]; and, --[0274] To track moving objects, in particular people, one or more embodiments of the system may incorporate a background subtraction or motion filter algorithm, masking out the background from the foreground for each of the planar projected images.--, in [0274]; and, --[0286] Appearance extraction from image 30G may for example be done by histograms, or by any other dimensionality reduction method. A lower dimensional vector may be formed from the composite image of each tracked person and used to compare it with other tracked subjects. For example, a neural network may be trained to take composite cylindrical images as input, and to output a lower-dimensional vector that is close to other vectors from the same person and far from vectors from other persons. To distinguish between people, vector-to-vector distances may be computed and compared to a threshold; for example, a distance of 0.0 to 0.5 may indicate the same person, and a greater distance may indicate different people. One or more embodiments may compare tracks of people by forming distributions of appearance vectors for each track, and comparing distributions using a distribution-to-distribution measure (such as KL-divergence, for example). A discriminant between distributions may be computed to label a new vector to an existing person in a store or site.--, in [0286], and, --[0368] During store operation, the quantity sensors may feed data into the signal processor 6610 which collects statistics on quantity measurements such as distance, weight, or other variables, and reports as a data packet of amount changed (distance/weight/other quantity variables) and time of start and end of the change.--, in [0368]); and
determine the motion distance based on the second location information and the third location information (see WEXLER : e.g., -- monitoring module 603 may be provided for continuous monitoring. Such continuous monitoring may include tracking a movement of at least a portion of an object included in one or more images captured by the image sensor…. monitoring an object or person captured by an image sensor 220 may include tracking movement of the object across the fields of view of the plurality of image sensors 220.--, in [0208]-[0209], and, -- the individual may be recognized based on other physical characteristics or traits. For example, the system may detect a body shape or posture of the individual, which may indicate an identity of the individual. Similarly, an individual may have particular gestures, mannerisms (e.g., movement of hands, facial movements, gait, typing or writing patterns, eye movements, or other bodily movements) that the system may use to identify the individual.--, in [0239]; and, -- the at least one processor may be programmed to execute a method comprising receiving a first motion signal indicative of an output of at least one of a first motion sensor or a first location sensor of a mobile device. For example, motion sensor (e.g., accelerometer 6950) of mobile device 120 may sense a motion or change in velocity or acceleration of mobile device 120. For example, user 100 may be carrying or wearing mobile device 120 (e.g., a smartphone) and may be walking, running, riding, and/or traveling in, for example, a land-based, sea-based, or airborne vehicle. Accelerometer 6950 may periodically or continuously generate signals representative of the detected motion or change in velocity or acceleration of mobile device 120.--, in [0838]-[0839], and, -- determining whether the mobile device and the wearable device share the one or more motion characteristics includes determining whether the first motion signal and the second motion signal differ relative to one or more thresholds. For example, it is contemplated that processor 210 may determine one or more differences between various parameters (e.g., positions, speeds, accelerations, velocities, directions of movement, etc., over one or more periods of time) associated with wearable device 110 and mobile device 120. It is contemplated that differences may be obtained in many ways, for example, vector distance, cosine distance, or by performing other mathematical operations known in the art for determining differences. By way of example, processor 210 may determine differences between the positions of wearable device 110 and mobile device 120 over a plurality of time periods. Furthermore, processor 210 may compare the determined differences with one or more thresholds. Processor 210 may determine that wearable device 110 and mobile device 120 share one or more motion characteristics when the corresponding differences are about zero, or are less than corresponding correlation thresholds.--, in [0851]; also see BUIBAS: e.g., --[0223] Various methods may be used to calculate a 3D field of influence volume around a person. FIGS. 6A through 6E illustrate a method that may be used in one or more embodiments. (These figures illustrate the construction of a field of influence volume using 2D figures, for ease of illustration, but the method may be applied in three dimensions to build a 3D volume around the person.) Based on an image or images 601 of a person, image analysis may be used to identify landmarks on the person's body. For example, landmark 602 may be the left elbow of the person. FIG. 6B illustrates an analysis process that identifies 18 different landmarks on the person's body. One or more embodiments may identify any number of landmarks on a body, at any desired level of detail. Landmarks may be connected in a skeleton in order to track the movement of the person's joints. Once landmark locations are identified in the 3D space associated with the store, one method for constructing a 3D field of influence volume is to calculate a sphere around each landmark with a radius of a specified threshold distance--, in [0223], --[0254] The processor 130 tracks person 1901 to location 1923 at time 1924, where credential reader 1905 is located. In one or more embodiments location 1923 may be the same as location 1921 where tracking begins; however, in one or more embodiments the person may be tracked in an area upon entering the area and may provide a credential at another time, such as upon entering or exiting a store. In one or more embodiments, multiple credential readers may be present; for example, the gas station in FIG. 19 may have several pay-at-the-pump stations at which customers can enter credentials. Using analysis of camera images, processor 130 may determine which credential reader a person uses to enter a credential, which allows the processor to associate an authorization with the person, as described below.
[0255] As a result of entering credential 1904 into credential reader 1905, an authorization 1907 is provided to gas pump 1902. This authorization, or related data, may also be transmitted to processor 130. The authorization may for example be sent as a message 1910 from the pump or credential reader, or directly from bank or payment processor (or another authorization service) 212. Processor 130 may associate this authorization with person 1901 by determining that the trajectory 1920 of the person is at or near the location of the credential reader 1904 at or near the time that the authorization message is received or the time that the credential is presented to the credential reader 1905. In embodiments with multiple credential readers in an area, the processor 130 may associate a particular authorization with a particular person by determining which credential reader that authorization is associated with and by correlating the time of that authorization and the location of that credential reader with the trajectories of one or more people to determine which person is at or near that credential reader at that time. In some situations, the person 1901 may wait at the credential reader 1905 until the authorization is received; therefore processor 130 may use either the time that the credential is presented or the time that the authorization is received to determine which person is associated with the authorization.
[0256] By determining that person 1901 is at or near location 1923 at or near time 1924, determining that location 1923 is the location of credential reader 1905 (or within a zone near the credential reader) and determining that authorization 1910 is associated with credential reader 1905 and is received at or near time 1924 (or is associated with presentation of a credential at or near time 1924), processor 130 may associate the authorization with the trajectory 1920 of person 1901 after time 1924. This association 1932 may for example add an extended tag 1933 to the trajectory that includes authorization information and may include account or credential information associated with the authorization. Processor 130 may also associate certain allowed actions with the authorization; these allowed actions may be specific to the application and may also be specific to the particular authorization obtained for each person or each credential.
[0257] Processor 130 then continues to track the trajectory 1920 of person 1901 to the location 1925 at time 1926.--, in [0254]-[0257]; and, --[0274] To track moving objects, in particular people, one or more embodiments of the system may incorporate a background subtraction or motion filter algorithm, masking out the background from the foreground for each of the planar projected images.--, in [0274]; and, --[0286] Appearance extraction from image 30G may for example be done by histograms, or by any other dimensionality reduction method. A lower dimensional vector may be formed from the composite image of each tracked person and used to compare it with other tracked subjects. For example, a neural network may be trained to take composite cylindrical images as input, and to output a lower-dimensional vector that is close to other vectors from the same person and far from vectors from other persons. To distinguish between people, vector-to-vector distances may be computed and compared to a threshold; for example, a distance of 0.0 to 0.5 may indicate the same person, and a greater distance may indicate different people. One or more embodiments may compare tracks of people by forming distributions of appearance vectors for each track, and comparing distributions using a distribution-to-distribution measure (such as KL-divergence, for example). A discriminant between distributions may be computed to label a new vector to an existing person in a store or site.--, in [0286], and, --[0368] During store operation, the quantity sensors may feed data into the signal processor 6610 which collects statistics on quantity measurements such as distance, weight, or other variables, and reports as a data packet of amount changed (distance/weight/other quantity variables) and time of start and end of the change.--, in [0368]).
Re Claim 7, WEXLER as modified by BUIBAS further disclose wherein the target object comprises an eyeball (see BUIBAS: e.g., --[0230] FIG. 13 also illustrates that the system has detected a “look at” action 1304 by shopper 1111 with respect to item 1202 that the shopper picked up. In one or more embodiments, the system may detect that a person is looking at an item by tracking the eyes of the person (as landmarks, for example) and by projecting a field of view from the eyes towards items. If an item is within the field of view of the eyes, then the person may be identified as looking at the item. For example, in FIG. 13 the field of view projected from the eyes landmarks of shopper 1111 is region 1305 and the system may recognize that item 1202 is within this region. One or more embodiments may detect that a person is looking at an item whether or not that item is moved by the person; for example, a person may look at an item in an item storage area while browsing and may subsequently choose not to touch the item. [0231] In one or more embodiments, other head landmarks instead of or in addition to the eyes may be used to compute head orientation relative to the store reference frame to determine what a person is looking at.--, in [0230]-[0231]).
Re Claims 8-14, claims 8-14 are corresponding method claim to claims 1-7, respectively. Claims 8-14 thus are rejected for the similar reasons for claims 1-7. See above discussions with regard to claims 1-7 respectively. WEXLER as modified by BUIBAS further disclose a method for tracking objects (see WEXLER : e.g., --the at least one tracking subsystem includes at least one processor programmed to: receive a plurality of images from the one or more cameras; identify at least one individual represented by the plurality of images; determine at least one characteristic of the at least one individual; and generate and send an alert based on the at least one characteristic.--, in abstract; and, -- tracking a movement of at least a portion of an object included in one or more images captured by the image sensor. For example, in one embodiment, apparatus 110 may track an object as long as the object remains substantially within the field-of-view of image sensor 220… monitoring an object or person captured by an image sensor 220 may include tracking movement of the object across the fields of view of the plurality of image sensors 220..--, in [0208]-[0209], and, -- the system may track statistical information associated with interactions with individuals. For example, the system may track interactions with each encountered individual and automatically update a personal record of interactions with the encountered individual. The system may provide analytics and tags per individual based on meeting context (e.g., work meeting, sports meeting, etc.). Information, such as a summary of the relationship, may be provided to the user via an interface. In some embodiments, the interface may order individuals chronologically based on analytics or tags. For example, the system may group or order individuals by attendees at recent meetings, meeting location, amount of time spent together, or various other characteristics. Accordingly, the disclosed embodiments may provide, among other advantages, improved efficiency, convenience, and functionality over prior art wearable apparatuses.--, in [0237]-[0238], and, -- processor 210 may compare one or more images of the captured plurality of images (or characteristics such as color, pattern, shapes, etc. in the captured images) to a database of images/characteristics stored in memory 550a of apparatus 110 (and/or memory 550b of computing device 120) to identify that the user is drinking coffee… processor 210 may detect the type of beverage that user 3610 is drinking and/or track the quantity of the beverage consumed by user 3610.--, in [0509], [0524]-[0525]).
Re Claims 15-20, claims 15-20 are corresponding medium claim to claims 1-6, respectively. Claims 15-20 thus are rejected for the similar reasons for claims 1-6. See above discussions with regard to claims 1-6 respectively. WEXLER as modified by BUIBAS further disclose a non-transitory computer-readable storage medium storing instructions, which when executed by a processor, cause the processor to perform a method for tracking objects (see WEXLER : e.g., --the at least one tracking subsystem includes at least one processor programmed to: receive a plurality of images from the one or more cameras; identify at least one individual represented by the plurality of images; determine at least one characteristic of the at least one individual; and generate and send an alert based on the at least one characteristic.--, in abstract; and, -- tracking a movement of at least a portion of an object included in one or more images captured by the image sensor. For example, in one embodiment, apparatus 110 may track an object as long as the object remains substantially within the field-of-view of image sensor 220… monitoring an object or person captured by an image sensor 220 may include tracking movement of the object across the fields of view of the plurality of image sensors 220..--, in [0208]-[0209], and, -- the system may track statistical information associated with interactions with individuals. For example, the system may track interactions with each encountered individual and automatically update a personal record of interactions with the encountered individual. The system may provide analytics and tags per individual based on meeting context (e.g., work meeting, sports meeting, etc.). Information, such as a summary of the relationship, may be provided to the user via an interface. In some embodiments, the interface may order individuals chronologically based on analytics or tags. For example, the system may group or order individuals by attendees at recent meetings, meeting location, amount of time spent together, or various other characteristics. Accordingly, the disclosed embodiments may provide, among other advantages, improved efficiency, convenience, and functionality over prior art wearable apparatuses.--, in [0237]-[0238], and, -- processor 210 may compare one or more images of the captured plurality of images (or characteristics such as color, pattern, shapes, etc. in the captured images) to a database of images/characteristics stored in memory 550a of apparatus 110 (and/or memory 550b of computing device 120) to identify that the user is drinking coffee… processor 210 may detect the type of beverage that user 3610 is drinking and/or track the quantity of the beverage consumed by user 3610.--, in [0509], [0524]-[0525]).
Conclusion
Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEI WEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on 8:00 - 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662
Prosecution Insights