DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Restriction
The Applicant respond the Restriction by electing claims 1-14, 19-20 and without traverse. Claims 15-18 are cancelled. Claims 21-24 are newly added.
Information Disclosure Statement
IDS is not filed.
Priority
Foreign priority is not claimed.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 9, 10, 11 and 12 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 9 recite limitations – “determining a first distance between the user and a centerline associated with a device…”, appears to be directed to calculating first distance between the user and a centerline of the device. However, it is not clear as to from which angle first and second images are captured to determine centerline between the user and a specific device and if there is camera sensor that collects such images or the device itself has in built camera to capture such images.
Remaining claims 10, 11 and 12 recite features that calculates various orientations and vectors based on if the user is in front or vectors to representing distance between the user’s head and side of a device, perpendicular to the device. These calculations assist in determining the user is in direct line of sight of the device in order to estimate user’s attentiveness and if it is the primary user of the application. However, it is not clear as to from which specific angles of cameras are used so that such images are captured in order to represent all angles including perpendicular, direct centerline, side lines etc. or if there are other sensors located within the device that calculates such distances and vectors.
Therefore, Claims 9-12 are rejected under 35 U.S.C. 112(b).
Therefore, the Examiner suggests to amend the claims in order to accurately define the discussed features in order to render the claims definite.
Dependent claims do not remedy the deficiencies introduced the independent claims are similarly rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-14, 19-22 and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (CA 2986860 A1, English Translation) in view of Buibas et al. (US Pub No. 20210158430 A1).
Regarding Claim 1,
Wang discloses A method comprising: determining, using one or more machine learning models and based at least on first image data representative of a first image associated with a first time, at least a first bounding shape associated with a head of a user and one or more first points associated with a body of the user; (Wang, [0007], [0015], [0053], three of those for binary face detection/classification, and three more for bounding box calibration. This face detection framework can be highly suitable for implementations in the embedded environments due to the cascade design and the simple CNN used by each stage. Note that, each of the bounding box calibration stages in the cascaded CNN requires an additional CNN and thus extra computational expense. Moreover, in the cascaded CNN, the inherent correlation between face detection and face alignment is ignored. integrates the face detection and face alignment operations using unified cascaded CNNs through multi-task learning. In principal, the MTCNN also uses several coarse-to-fine CNN stages to operate on different resolutions of the input image. However, in the MTCNN, facial landmark localization, binary face classification, and bounding box calibration are trained jointly using a single CNN in each stage. As a result, only three stages are needed in the MTCNN; discloses the first processing module processes the detected candidate image region using the first image size by: for each image in the multi-resolution representations of the detected candidate image region, applying a sliding window of the first image size to generate a set of image patches for the image; and generating the first set of candidate facial windows from each set of the image patches corresponding to the locations of the sliding window within each of the multi-resolution representations of the detected candidate image region; to improve real-time face detection efficiency, the proposed face detection technique and system can also identify the facial landmark points of each detected face (such as eyes, noses, and mouths). This information allows the system to track each face, select the best-pose image (also referred to as "the best face") of each person, e.g., the one that is closest to the front-view, and send the best face to the server for further processing, such as face retrieval. By transmitting the faces in the video frame, it is not necessary to transmit the entire video frame to the server for some application face region in first image is detected with bounding box and landmark points are also obtained to predict candidate regions)
determining, based at least on at least one of the first bounding shape and the one or more first points, at least a predicted bounding shape associated with the user; (Wang, [0015-0016], [0078], discloses the first processing module processes the detected candidate image region using the first image size by: for each image in the multi-resolution representations of the detected candidate image region, applying a sliding window of the first image size to generate a set of image patches for the image; and generating the first set of candidate facial windows from each set of the image patches corresponding to the locations of the sliding window within each of the multi-resolution representations of the detected candidate image region; each of the first set of candidate facial windows is associated with a confident score and a set of bounding box coordinates defining the location of the candidate facial window within the detected candidate image region; describe each of the blocks in face detection system 300 in more detail; input video image 302 is first received by motion. Detection module 304. In some embodiments, it is assumed that a human face within a given video is associated with a motion. Hence, to reduce the computational complexity, motion detection module 304 can be used to locate and identify those areas within each video frame which are associated with motions based on comparisons with previously received video frames. Note that these moving areas can include both human objects and non-human objects such as a moving vehicle. Moreover, for a moving human object, a moving area can include both the human face and the human body; bounding box as candidate region is estimated as predicted bounding box)
determining, using the one or more machine learning models and based at last on second image data representative of a second image associated with a second time, at least a second bounding shape associated with the head of the user and; (Wang, [0088-0089], discloses in the final 16x1 output vector, the first 2 outputs are used to generate the face detection confidence score (also referred as to as face classifier); the next 4 outputs are the bounding box coordinates (also referred as to as bounding box regression operator) of the face in the image patch 402 (i.e., if a face is detected in the image patch 402); and the last 10 outputs indicate the locations of 5 facial landmark points of the detected face, i.e., left eye, right eye, nose, and two corners of the mouth (also referred as to as landmark localization operator). Hence, first stage CNN 400 outputs a set of candidate facial windows/bounding boxes corresponding to a subset of the image patches 320 shown in Fig. 3; multiple head shape and associated landmark points in second image is determined as second bound shape) and
Wang does not explicitly disclose one or more second points associated with the body of the user; determining, based at least on the predicted bounding shape, the second bounding shape, and the one or more second points, that the user depicted in the second image corresponds to the user depicted in the first image.
Buibas discloses one or more second points associated with the body of the user; determining, based at least on the predicted bounding shape, the second bounding shape, and the one or more second points, that the user depicted in the second image corresponds to the user depicted in the first image. (Buibas, [0207], discloses people in the store may be tracked as they move through the store. Since multiple people may be moving in the store simultaneously, it may be beneficial to distinguish between persons using image analysis, so that people can be correctly tracked. FIG. 5 shows an illustrative method that may be used to distinguish among different persons. As a new person 501 enters a store or enters a specified area or areas of the store at time 510, images of the person from cameras such as cameras 511, 512 and 513 may be analyzed to determine certain characteristics 531 of the person's appearance that can be used to distinguish that person from other people in the store. These distinguishing characteristics may include for example, without limitation: the size or shape of certain body parts; the color, shape, style, or size of the person's hair; distances between selected landmarks on the person's body or clothing; the color, texture, materials, style, size, or type of the person's clothing, jewelry, accessories, or possessions; the type of gait the person uses when walking or moving; the speed or motion the person makes with any part of their body such as hands, arms, legs, or head; and gestures the person makes. One or more embodiments may use high resolution camera images to observe biometric information such as a person's fingerprints or handprints, retina, or other features; features including person’s physical features are extracted and processed as second image points to determine second bound shape in second image to determine same person is determined based on tracking and analyzing trajectory of pose of body and facial regions in multiple first and second images captured)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Wang in view of Buibas having a method of detecting face of a user using shape and landmark points as candidate region bound boxes in plurality of images, with the teachings of Buibas having, by the module, determining trajectory of persons and distinguishing their shape and size or other features to determine the identity of a person (user) if it is the same person tracked previously in first frame to second frame in order to accurately identify person in consecutive frame in applications including retail shopping.
Regarding Claim 2,
The combination of Wang and Buibas further discloses determining, using the one or more machine learning models and based at least on the first image data, a third bounding shape associated with a face of the user; and determining, using the one or more machine learning models and based at least on the second image data, a fourth bounding shape associated with the face of the user, wherein the determining the predicted bounding shape is further based at least on the third bounding shape, and wherein the determining the user depicted in the second image corresponds to the user depicted in the first image is further based at least on the fourth bounding shape. (Wang, [0007], discloses 100081 In the MTCNN, a multi-task cascaded CNN is proposed, which integrates the face detection and face alignment operations using unified cascaded CNNs through multi-task learning. In principal, the MTCNN also uses several coarse-to-fine CNN stages to operate on different resolutions of the input image. However, in the MTCNN, facial landmark localization, binary face classification, and bounding box calibration are trained jointly using a single CNN in each stage. As a result, only three stages are needed in the MTCNN. More specifically, the first stage of the MTCNN generates candidate facial windows quickly through a shallow CNN. Next, the second stage of the MTCNN refines the candidate windows by rejecting a large number of non-face windows through a more complex CNN. Finally, the third stage of the MTCNN uses a more powerful CNN to further decide whether each input window is a face or not. If it is determined to be so, the locations of five facial landmarks are also estimated. The performance of the MTCNN is notably improved compared to previous face detection systems. The MTCNN framework is generally more suitable for implementations on resource-limited embedded systems compared to the aforementioned cascaded CNN framework; course to fine face detection is performed using machine learning CNN model that detects face in multiple times from course to fine detection of face of a user is detected from the set of images captured). Additionally, the rational and motivation to combine the references Wang and Buibas as applied in rejection of claim 1 apply to this claim.
Regarding Claim 4,
The combination of Wang and Buibas further discloses determining a third bounding shape based at least on the second bounding shape and the one or more second points; determining an amount of overlap between the predicted bounding shape and the third bounding shape; and determining, based at least on the amount of overlap, that the user depicted in the second image corresponds to the user depicted in the first image. (Wang, [ 0092-0093], discloses as a result of generating multi-resolution representations using the pyramid technique and image patches with the sliding window technique, many or multiple overlapping but non-identical bounding boxes can be generated around each face in the input image. In some embodiments, for each image block that is classified by first stage CNN 308 as a face, a corresponding image region in the original input video image 302 is identified. Next, a non-maximum suppression (NMS) technique can be used to merge those highly overlapped bounding boxes, as described in the MTCNN. Note that this NMS operation can be performed after the aforementioned filtering operation on the candidate facial windows. In some embodiments, the NMS operation is implemented in first stage CNN 308 within face detection system 300. After the NMS operation, the remaining bounding boxes can be refined through a bounding box regression operation to refine the bounding box locations, as also described in the MTCNN. Again, this NMS operation can be implemented in first stage CNN 308 within face detection system 300. Hence, after one or more additional processing, first stage CNN 308 outputs a set of candidate bounding boxes of faces, or candidate facial windows (Estimated bounding shape) for each candidate facial window 322 output by first stage CNN 308, a corresponding image patch is located and cropped from the original input video image 302, and the cropped image patch is subsequently resized to a user- specified input size of second stage CNN 310; multiple overlapping shapes are obtained from sub images from which candidate regions (first, second, third, fourth bound boxes) of user facial images are predicted as candidate facial windows and overlapping shapes are merged and determined as user face image). (Buibas, [0207], discloses since multiple people may be moving in the store simultaneously, it may be beneficial to distinguish between persons using image analysis, so that people can be correctly tracked. FIG. 5 shows an illustrative method that may be used to distinguish among different persons. As a new person 501 enters a store or enters a specified area or areas of the store at time 510, images of the person from cameras such as cameras 511, 512 and 513 may be analyzed to determine certain characteristics 531 of the person's appearance that can be used to distinguish that person from other people in the store. These distinguishing characteristics may include for example, without limitation: the size or shape of certain body parts; the color, shape, style, or size of the person's hair; distances between selected landmarks on the person's body or clothing; the color, texture, materials, style, size, or type of the person's clothing, jewelry, accessories, or possessions; the type of gait the person uses when walking or moving; the speed or motion the person makes with any part of their body such as hands, arms, legs, or head; and gestures the person makes. One or more embodiments may use high resolution camera images to observe biometric information such as a person's fingerprints or handprints, retina, or other features; same person is determined based on tracking and analyzing trajectory of pose of body and facial regions in multiple first and second images captured) (Additionally, the rational and motivation to combine the references Wang and Buibas as applied in rejection of claim 1 apply to this claim.
Regarding Claim 5,
The combination of Wang and Buibas further discloses determining, based at least on at least a portion of the one or more first points, a third bounding shape; determining, based at least on the third bounding shape and the first bounding shape, that the first bounding shape is associated with the one or more first points; and associating, based at least on the first bounding shape being associated with the one or more first points, the first bounding shape and the one or more first points with an identifier associated with the user. (Buibas, [0207], discloses since multiple people may be moving in the store simultaneously, it may be beneficial to distinguish between persons using image analysis, so that people can be correctly tracked. FIG. 5 shows an illustrative method that may be used to distinguish among different persons. As a new person 501 enters a store or enters a specified area or areas of the store at time 510, images of the person from cameras such as cameras 511, 512 and 513 may be analyzed to determine certain characteristics 531 of the person's appearance that can be used to distinguish that person from other people in the store. These distinguishing characteristics may include for example, without limitation: the size or shape of certain body parts; the color, shape, style, or size of the person's hair; distances between selected landmarks on the person's body or clothing; the color, texture, materials, style, size, or type of the person's clothing, jewelry, accessories, or possessions; the type of gait the person uses when walking or moving; the speed or motion the person makes with any part of their body such as hands, arms, legs, or head; and gestures the person makes. One or more embodiments may use high resolution camera images to observe biometric information such as a person's fingerprints or handprints, retina, or other features; same person is determined based on tracking and analyzing trajectory of pose of body and facial regions in multiple first and second images captured) (Wang,[ 0092-0093], discloses as a result of generating multi-resolution representations using the pyramid technique and image patches with the sliding window technique, many or multiple overlapping but non-identical bounding boxes can be generated around each face in the input image. In some embodiments, for each image block that is classified by first stage CNN 308 as a face, a corresponding image region in the original input video image 302 is identified. Next, a non-maximum suppression (NMS) technique can be used to merge those highly overlapped bounding boxes, as described in the MTCNN. Note that this NMS operation can be performed after the aforementioned filtering operation on the candidate facial windows. In some embodiments, the NMS operation is implemented in first stage CNN 308 within face detection system 300. After the NMS operation, the remaining bounding boxes can be refined through a bounding box regression operation to refine the bounding box locations, as also described in the MTCNN. Again, this NMS operation can be implemented in first stage CNN 308 within face detection system 300. Hence, after one or more additional processing, first stage CNN 308 outputs a set of candidate bounding boxes of faces, or candidate facial windows (Estimated bounding shape) for each candidate facial window 322 output by first stage CNN 308, a corresponding image patch is located and cropped from the original input video image 302, and the cropped image patch is subsequently resized to a user- specified input size of second stage CNN 310; multiple overlapping shapes are obtained from sub images from which candidate regions (first, second, third, fourth bound boxes) of user facial images are predicted as candidate facial windows and overlapping shapes are merged and determined as user face image). Additionally, the rational and motivation to combine the references Wang and Buibas as applied in rejection of claim 1 apply to this claim.
Regarding Claim 6,
The combination of Wang and Buibas further discloses determining, based at least on at least one of the second bounding shape or the one or more second points, a two-dimensional location associated with the user; and determining, based at least on the two-dimensional location, a three-dimensional location associated with the user. (Wang,[ 0092-0093], discloses as a result of generating multi-resolution representations using the pyramid technique and image patches with the sliding window technique, many or multiple overlapping but non-identical bounding boxes can be generated around each face in the input image. In some embodiments, for each image block that is classified by first stage CNN 308 as a face, a corresponding image region in the original input video image 302 is identified. Next, a non-maximum suppression (NMS) technique can be used to merge those highly overlapped bounding boxes, as described in the MTCNN. Note that this NMS operation can be performed after the aforementioned filtering operation on the candidate facial windows. In some embodiments, the NMS operation is implemented in first stage CNN 308 within face detection system 300. After the NMS operation, the remaining bounding boxes can be refined through a bounding box regression operation to refine the bounding box locations, as also described in the MTCNN. Again, this NMS operation can be implemented in first stage CNN 308 within face detection system 300. Hence, after one or more additional processing, first stage CNN 308 outputs a set of candidate bounding boxes of faces, or candidate facial windows (Estimated bounding shape) for each candidate facial window 322 output by first stage CNN 308, a corresponding image patch is located and cropped from the original input video image 302, and the cropped image patch is subsequently resized to a user- specified input size of second stage CNN 310; multiple overlapping shapes are obtained from sub images from which candidate regions (first, second, third, fourth bound boxes) of user facial images are predicted as candidate facial windows and overlapping shapes are merged and determined as user face image). Additionally, the rational and motivation to combine the references Wang and Buibas as applied in rejection of claim 1 apply to this claim.
Regarding Claim 7,
The combination of Wang and Buibas further discloses determining, based at least on the second bounding shape, a first two-dimensional location associated with the user; determining, based at least on the one or more second points, a second two- dimensional location associated with the user; determining, based at least on the first two-dimensional location, a first three-dimensional location associated with the user; determining, based at least on the second two-dimensional location, a second three- dimensional location associated with the user; and determining, based at least on the first three-dimensional location and the second three-dimensional location, a final three-dimensional location associated with the user. (Wang, [0071-0074], discloses the array of feature maps 206 includes a set of three-dimensional (3D) matrices (i.e., two dimensions for a given feature map and one dimension for the number of feature maps). For example, the array of feature maps 206 can be composed of nine (i.e., a 3x3 array of) 3D matrices of 2x2x48 sizes, wherein nine is the number of subimages 204 having indices of 0, 1, 2, ..., 8 (i.e., subimages of 3 rows by 3 columns), 2x2 is the size of a single output feature map after CNN1 module 214 for each subimage, and 48 is the number of feature maps for each subimage. In some embodiments, merging module 222 is configured to merge the array of feature maps 206 by concatenating the set of 3D output matrices based on the corresponding indices to form a merged 3D feature-map matrix, while preserving the spatial relationships of the set of subimages 204. In the above example, this step generates a 3D matrix of 6x6x48. Next, the merged 3D matrix can be flattened into a one-dimensional (1D) vector. In the above example, this creates a 1D vector having a size of 1728. Finally, the flattened 1D vector is fed into the second processing stage 224; Fig. 2B shows that the merged feature maps 208 generated by merging module 222 are fed into the second processing stage 224 of CNN system 210 for further processing; Of the two aforementioned face detection frameworks, the MTCNN has a simpler structure than the cascaded CNN because the MTCNN uses three CNN stages compared to the six stages used by the cascaded CNN. Moreover, the MTCNN can detect the facial landmark locations, which are useful to track a person and decide the pose of each face. Consequently, various examples of the proposed face detection CNN system and technique described below are based on the MTCNN framework which uses three stages. However, it should be noted that the proposed face detection CNN system and technique can also be applied to the cascaded CNN framework; three-dimensional location of user is determined based on two dimensional feature map of user locations in first, second and multiple bound boxes). Additionally, the rational and motivation to combine the references Wang and Buibas as applied in rejection of claim 1 apply to this claim.
Regarding Claim 8,
The combination of Wang and Buibas further discloses determining, based at least on at least one of the second bounding shape or the one or more second points, a zone, from a plurality of zones, that the user is located within; and outputting data based at least on the zone. (Buibas, [0021], discloses the processor identifies an item storage area that is proximal to the trajectory of the person during an interaction time period utilizes a 3D location of the storage area that intersects the 3D field of influence volume around the person during the interaction time period. In one or more embodiments, the processor calculates the 3D field of influence volume around the person utilizing a spatial probability distribution for multiple landmarks on the person at the points of time during the time period, wherein each landmark of the multiple landmarks corresponds to a location on a body part of the person. In one or more embodiments, the 3D field of influence volume around the person comprises points having a distance to a closest landmark of the multiple landmarks that is less than or equal to a threshold distance. In one or more embodiments, the 3D field of influence volume around the person comprises a union of probable zones for each landmark of the multiple landmarks, wherein each probable zone of the probable zones contains a threshold probability of the spatial probability distribution for a corresponding landmark. In one or more embodiments, the processor calculates the spatial probability distribution for multiple landmarks on the person at the points of time during the time period through calculation of a predicated spatial probability distribution for the multiple landmarks at one or more points of time during the time period based on a physics model and calculation of a corrected spatial probability distribution at one or more points of time during the time period based on observations of one or more of the multiple landmarks in the time sequence of images. In one or more embodiments, the physics model includes the locations and velocities of the landmarks and thus the calculated field of influence. This information can be used to predict a state of landmarks associated with a field at a time and a space not directly observed and thus may be utilized to interpolate or augment the observed landmarks; probable person zones are determined from image) (Wang, [0052], discloses to improve real-time face detection performance, the proposed face detection technique and system detects those moving areas in each of the video frames/images. For example, the proposed face detection technique and system can use a built-in background subtraction module of Hi3519 to detect those moving areas in the video frame. Next, the proposed face detection technique and system uses a coarse-to-fine multi-stage CNN to detect most or all faces in the video frame. More specifically, for each stage in the multi-stage CNN which has an input image size constraint, the subimage-based CNN framework can be applied. For example, some embodiments of the proposed face detection technique only need to apply the subimage-based CNN framework to the last stage of a multi-stage CNN framework; moving areas (zones) are detected of user location in captured images). Additionally, the rational and motivation to combine the references Wang and Buibas as applied in rejection of claim 1 apply to this claim.
Regarding Claim 9,
The combination of Wang and Buibas further discloses determining a first distance between the user and a centerline associated with a device and a second distance between a second user and the centerline associated with the device; and
determining, based at least on the first distance and the second distance, that the user is a primary user associated with the device. (Buibas, [0019-0021
], Fig. 2, discloses each image of the time sequence of images is a 2D image and the processor calculates a trajectory of the person consisting of a 3D location and orienation of the person and at least one body landmark from two or more 2D projections of the person in the time sequence of images; orientation and distance from front camera is determined in captured images of the user (person) using the camera device and fig. 1 discloses cameras 121 and/0r 122 and user 103 from difference line of sight from centerline of camera and determined as same primary person based on location in three dimensional space). Additionally, the rational and motivation to combine the references Wang and Buibas as applied in rejection of claim 1 apply to this claim.
Regarding Claim 10,
The combination of Wang and Buibas further discloses determining, based at least on the user being the primary user and using at least one of the first image data or the second image data, one or more attributes associated with the user. (Buibas, [0051] In one or more embodiments the projected images may be analyzed by inputting them into a machine learning system that outputs an intensity map that contains a likelihood that a person is at each location. The machine learning system may be a convolutional neural network, for example. An illustrative neural network architecture that may be used in one or more embodiments is a first half subnetwork consisting of copies of a feature extraction network, one copy for each projected image, a feature merging layer that combines outputs from the copies of the feature extraction network, and a second half subnetwork that maps combined features into the intensity map; user features (attributes) are processed from obtained first, second and multiple images to determine location of user (person) in the images). Additionally, the rational and motivation to combine the references Wang and Buibas as applied in rejection of claim 1 apply to this claim.
Regarding Claim 11,
The combination of Wang and Buibas further discloses determining that the user is located in front of a device; based at least on the user being located in front of the device, determining a first vector that is perpendicular to the device; determining a second vector that is associated with an orientation of the user; and determining, based at least on the first vector and the second vector, an attentiveness of the user with respect to the device. (Buibas, [0019], [0275], [0362-0363], [0019], Fig. 1, discloses each image of the time sequence of images is a 2D image and the processor calculates a trajectory of the person consisting of a 3D location and orientation of the person and at least one body landmark from two or more 2D projections of the person in the time sequence of images; appearance extraction from image 30G may for example be done by histograms, or by any other dimensionality reduction method. A lower dimensional vector may be formed from the composite image of each tracked person and used to compare it with other tracked subjects. For example, a neural network may be trained to take composite cylindrical images as input, and to output a lower-dimensional vector that is close to other vectors from the same person and far from vectors from other persons. To distinguish between people, vector-to-vector distances may be computed and compared to a threshold; for example, a distance of 0.0 to 0.5 may indicate the same person, and a greater distance may indicate different people. One or more embodiments may compare tracks of people by forming distributions of appearance vectors for each track, and comparing distributions using a distribution-to-distribution measure (such as KL-divergence, for example). A discriminant between distributions may be computed to label a new vector to an existing person in a store or site; data from sensors in the store, including for example, without limitation, cameras 7002a and 7002b and sensors 7004a and 7005a in item storage areas, may be transmitted to one or more processors 130 for analysis 7020. The result of this analysis may include information 7021 describing shopper activity. This information 7021 may include an activity history for each person that is detected in the store. A shopper's activity history may for example include the time period during which the shopper is in the store, the trajectory of the shopper through the store (which may associate each time in that time period with a location), and the actions taken by the shopper to interact with items or item storage areas. Illustrative table 7022 for example has a series of entries that contain an identifier 7022a of the shopper, a date and time 7022b when the shopper performed an action, position coordinates 7022c within the store where the action occurred, and the type of event 7022d associated with the action. Only selected events are shown in table 7022; in practice the shopper activity history for each shopper may contain hundreds or thousands of events, which may be sampled for example at regular intervals such as once per second, or recorded when sensor data indicates specific state changes or actions. This table is illustrative; one or more embodiments may use any type of data structure to describe and track shopper activity; various vectors associated with user’s location in three dimensional with respect to the camera device is calculated and vectors are derived to determine user’s location from captured images). Additionally, the rational and motivation to combine the references Wang and Buibas as applied in rejection of claim 1 apply to this claim.
Regarding Claim 12,
The combination of Wang and Buibas further discloses determining that the user is located outside of an area in front of a device; based at least on the user being located outside of the area in front of the device, determining a first vector that connects a side of the device to a head of the user; determining a second vector associated with an orientation of the user; and determining, based at least on the first vector and the second vector, an attentiveness of the user with respect to the device. (Buibas, [0207], people in the store may be tracked as they move through the store. Since multiple people may be moving in the store simultaneously, it may be beneficial to distinguish between persons using image analysis, so that people can be correctly tracked. FIG. 5 shows an illustrative method that may be used to distinguish among different persons. As a new person 501 enters a store or enters a specified area or areas of the store at time 510, images of the person from cameras such as cameras 511, 512 and 513 may be analyzed to determine certain characteristics 531 of the person's appearance that can be used to distinguish that person from other people in the store. These distinguishing characteristics may include for example, without limitation: the size or shape of certain body parts; the color, shape, style, or size of the person's hair; distances between selected landmarks on the person's body or clothing; the color, texture, materials, style, size, or type of the person's clothing, jewelry, accessories, or possessions; the type of gait the person uses when walking or moving; the speed or motion the person makes with any part of their body such as hands, arms, legs, or head; and gestures the person makes. One or more embodiments may use high resolution camera images to observe biometric information such as a person's fingerprints or handprints, retina, or other features; person (user) entering and exiting frames are tracked by if they are inside the frame or outside and their orientations are determined to estimate its position with respect to camera sensors). Additionally, the rational and motivation to combine the references Wang and Buibas as applied in rejection of claim 1 apply to this claim.
Regarding Claim 13,
The combination of Wang and Buibas further discloses storing data representative of a track associated with the user; and based at least on the determining the user depicted in the second image corresponds to the user depicted in the first image, updating the track associated with the user. (Buibas, [0215], discloses tracking a person as he or she moves through the store, one or more embodiments of the system may generate a 3D trajectory of the person through the store. This 3D trajectory may be combined with information on movement of items in item storage areas to associate people with the items they interact with. If the person's trajectory is proximal to the item at a time when the item is moved, then the movement of the item may be attributed to that person, for example. FIG. 9 illustrates this process. For ease of illustration, the person's trajectory and the item position are shown in two dimensions; one or more embodiments may perform a similar analysis in three dimensions using the 3D model of the store, for example. A trajectory 901 of a person is tracked over time, using a tracking process such as the one illustrated in FIG. 8, for example. For each person, a 3D field of influence volume 902 may be calculated at each point in time, based for example on the location or probability distribution of landmarks on the person's body. (Again, for ease of illustration the field of influence volume shown in FIG. 9 is in the two dimension, although in implementation this volume may be three dimensional.) The system calculates the trajectory of the 3D influence volume through the store. Using camera image analysis such as the analysis illustrated in FIG. 3, motion 903 of an item is detected at a location 904. Since there may be multiple people tracked in a store, the motion may be attributed to the person whose field of influence volume was at or near this location at the time of motion. Trajectory 901 shows that the field of influence volume of this tracked person intersected the location of the moved item during a time interval proximal in time to this motion; hence the item movement may be attributed to this person; user is tracked from first to second frame). Additionally, the rational and motivation to combine the references Wang and Buibas as applied in rejection of claim 1 apply to this claim.
Regarding Claim 14,
The combination of Wang and Buibas further discloses determining one or more criteria associated with the user, the one or more criteria including one or more of a distance to the user, a resolution associated with the user, or a velocity of associated with the user; and determining, based at least on the one or more criteria, one or more attributes associated with the user. (Buibas, [0207], [0278], discloses for example spherical models for heads, additional cylindrical models for upper and lower arms and/or upper and lower legs as well. These embodiments enable more detailed differentiation of users, and may be utilized in combination with gait analysis, speed of movement, any derivative of position, including velocity acceleration, jerk or any other frequencies of movement to differentiate users and their distinguishing characteristics. In one or more embodiments, the complexity of the model may be altered over time or as needed based on the number of users in a given area for example. Other embodiments may utilize simple cylindrical or other geometrical shapes per user based on the available computing power or other factors, including the acceptable error rate for example; people in the store may be tracked as they move through the store. Since multiple people may be moving in the store simultaneously, it may be beneficial to distinguish between persons using image analysis, so that people can be correctly tracked. FIG. 5 shows an illustrative method that may be used to distinguish among different persons. As a new person 501 enters a store or enters a specified area or areas of the store at time 510, images of the person from cameras such as cameras 511, 512 and 513 may be analyzed to determine certain characteristics 531 of the person's appearance that can be used to distinguish that person from other people in the store. These distinguishing characteristics may include for example, without limitation: the size or shape of certain body parts; the color, shape, style, or size of the person's hair; distances between selected landmarks on the person's body or clothing; the color, texture, materials, style, size, or type of the person's clothing, jewelry, accessories, or possessions; the type of gait the person uses when walking or moving; the speed or motion the person makes with any part of their body such as hands, arms, legs, or head; and gestures the person makes. One or more embodiments may use high resolution camera images to observe biometric information such as a person's fingerprints or handprints, retina, or other features; user’s features (attributes) are determined and tracked in images). Additionally, the rational and motivation to combine the references Wang and Buibas as applied in rejection of claim 1 apply to this claim.
Claim 19 recite processors with elements corresponding to the method steps recited in Claim 1. Therefore, the recited elements of the processor claim 19 are mapped to the proposed combination in the same manner as the corresponding steps of Claim 1. Additionally, the rationale and motivation to combine the Wang and Buibas references presented in rejection of Claim 1, apply to these claims.
Furthermore, the combination of Wang Buibas further discloses One or more processors comprising processing circuitry (Wang, [0027-0038], discloses an embedded system capable of performing face detection on captured video images is disclosed. This embedded system includes: a processor; a memory coupled to the processor; an image capturing device coupled to the processor and the memory and configured to capture video images; a hardware CNN module coupled to the processor and the memory; and a motion detection module coupled to the hardware CNN module and configured to receive a captured video image and detect a candidate image region corresponding to a moving object within the video image).
Regarding Claim 20,
The combination of Wang and Buibas further discloses wherein the one or more processors are comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more large language models (LLMs);a system for hosting or presenting one or more digital avatars; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs);a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources. (Wang, [0027-0038], discloses an embedded system capable of performing face detection on captured video images is disclosed. This embedded system includes: a processor; a memory coupled to the processor; an image capturing device coupled to the processor and the memory and configured to capture video images; a hardware CNN module coupled to the processor and the memory; and a motion detection module coupled to the hardware CNN module and configured to receive a captured video image and detect a candidate image region corresponding to a moving object within the video image).
Claim 19 recite device with procesors with elements corresponding to the method steps recited in Claim 1. Therefore, the recited elements of the processor claim 19 are mapped to the proposed combination in the same manner as the corresponding steps of Claim 1. Additionally, the rationale and motivation to combine the Wang and Buibas references presented in rejection of Claim 1, apply to these claims.
Claims 21-22 and 24 recite system with elements corresponding to the method steps recited in Claims 1, 2 and elements recited in claim 20. Therefore, the recited elements of the system claims 21-22 and 24 are mapped to the proposed combination in the same manner as the corresponding steps of Claims 1-2 and elements in claim 20. Additionally, the rationale and motivation to combine the Wang and Buibas references presented in rejection of Claim 1, apply to these claims.
Furthermore, the combination of Wang and Buibas further discloses A system comprising:one or more processors(Wang, [0027-0038], discloses an embedded system capable of performing face detection on captured video images is disclosed. This embedded system includes: a processor; a memory coupled to the processor; an image capturing device coupled to the processor and the memory and configured to capture video images; a hardware CNN module coupled to the processor and the memory; and a motion detection module coupled to the hardware CNN module and configured to receive a captured video image and detect a candidate image region corresponding to a moving object within the video image).
Allowable Subject Matter
Claims 3 and 23 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Han et al. (US Pub No. 20180121762 A, method involves determining context information by passing a layer output of a detection layer to another detection layer by multiple processors. The layer output of the former detection layer into the layer output of the lateral detection layer is incorporated. Context information is determined based on detecting a portion of object of interest. Object of interest from a portion of the object of interest is identified by each processor, where the object of interest includes multiple bounding boxes using set of image representation layers of a multilayer object model)
US-9996981-B1 (Tran, Systems and methods are disclosed for recommending products or services by receiving a 3D model of a product; capturing a reference object with a predetermined dimension in an environment where the product is to be placed using a mobile camera; determining one more dimensions of the environment relative to the predetermined dimension of the reference object; scaling the 3D model of the product based on dimensions of the environment and the product; and generating an augmented or virtual reality display of the product in the environment)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PINALBEN V PATEL whose telephone number is (571)270-5872. The examiner can normally be reached M-F: 10am - 8pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at 571-272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Pinalben Patel/Examiner, Art Unit 2673
Prosecution Insights