DETAILED ACTION
Claims 1-20 are pending in this application. Claims 1, 8, and 15 are currently amended.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/27/2023, 02/08/2024, 03/15/2024, 06/18/2024, 12/31/2024 and 02/06/2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 11/25/2025 has been entered.
Response to Arguments
35 U.S.C. 103
Applicant’s arguments (See Remarks filed 03/09/2025) have been fully considered by the Examiner and are persuasive. However, given the change in scope to the claims, a new grounds of rejection is made over Siddiquie, Zhou, and Zhang in view of Fisher as fully discussed below.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
1. Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Siddiquie (US 11396306 B1) in view of Zhou (US 10395385 B2), and in further view of Zhang (US 20130182114 A1) and Fisher (US 10055853 B2).
Regarding claim 1 Siddiquie discloses; A system comprising:
a plurality of sensors (Siddiquie, Column 14 lines 47-67 has disclosed that the image processing device has a computer, configured with a plurality of imaging devices), each sensor of the plurality of sensors configured to generate images of at least a portion of a space (Siddiquie Column 2, lines 5-35, one or more external imaging devices to capture an area, Figures 1a-f show the camera’s views for identifying and re-identifying a person in a store);
and a tracking subsystem communicatively coupled to the plurality of sensors (Siddiquie, Column 1, Lines 20-31, the system may include one or more sensors for the capture of image data, Column 11, the system has a re-identification system for tracking agents in the facility), the tracking subsystem comprising a processor configured to (Siddiquie, Column 3, Lines 55-66, the system has a processor unit (134) which may receive images):
receive the images generated by the plurality of sensors (Siddiquie, Column 3, Lines 55-66, the system has a processor unit (134) which may receive images);
track a first object and one or more other objects in the space using at least a portion of the images generated by the plurality of sensors (Siddiquie, Column 3, Lines 42-53, Locations of agents may be tracked via a “tracklet” which supplies the position of multiple agents via positions detection in image frames),
wherein the first object is associated with a first contour at a first depth and at least one of the other objects in the space is associated with a second contour at the first depth (Siddiquie, column 4 lines 38-52, images collected can be depth images, whereas per column 3 line 54 through column 4 line 37 the objects may be distinguished using background and foreground segmentations and other background vs foreground identification methods to distinguish objects to be tracked vs not to be tracked, the Examiner is interpreting this as depth information associated with the contour), wherein each of the first contour and the second contour comprises a curve associated with an edge of a representation of the respective first object or the at least one other object (Siddiquie (col 8 lines 20-35) discloses using a plurality of digital imaging devices to capture overlapping fields of view of data in a facility (col 2 lines 7-37). The image devices supplying image data to track users in a facility using the disclosed computing device of at least Fig 4 (“tracklet” of the one or more agents – col 2 lines 7-37). Siddique further disclosed (Fig 1H and Fig 1i items 180-1, 180-2, 180-3, 180-4, depicted as tracked rectangular blocks) two-person type objects being tracked, wherein each object is tracked based on contours instead of the blocks depicted in the figures (col 10 lines 19-40). The objects having a same or at least overlapping position (see Fig 1H and Fig 1i items 180-1 and overlapping/merged item 180-2, depicted as tracked rectangular blocks), wherein the determination that a same or overlapping object region corresponds to a request for object re-identification (a low confidence condition is initiated for proximate/merged/overlapping objects – col 19 lines 40-53, identified objects/agents of low confidence used for re-identification: col 19 lines 50-67, Column 10, lines 20-51, the system generates contours, shapes or outlines of the objects being tracked);
receive, from a first sensor of the plurality of sensors, a first image of the tracked first object (Siddiquie, column 2, lines 5-36, the system has a plurality of imaging devices set to capture a plurality of images of objects and agents, indicating at least a first and a second tracked object);
[determine, based on depth data associated with the first image at the first depth, that the first contour associated with the first object has merged with the second contour associated with the at least one other object to form a single contour;]
determine, from the first image, that the first object and the at least one other object have crossed a first predefined zone, wherein the first predefined zone identifies an area associated with a rack (Siddiquie, column 3 line 54- column 4 lines 36 teaches that objects may be tracked using bounding boxes (zones), and in column 10 line 20-51 the objects may be labeled and tracked when in motion, column 11 lines 10-32 and column 12 lines 10-33, items in an inventory at a store may be tracked, and the moved position and distance moved of the item are tracked, which indicates that the item is moving between “zones” and is tracked/detected. Figures 5a and 5b show this as an overhead where all the items are in a grid system, or “zones” and are tracked);
[determine that re-identification of the tracked first object is needed based at least in part upon the determination that the first contour merged with the second contour to form the single contour;
in response to determining that re-identification of the tracked first object is needed;
determine, based on the depth data associated with the first image at a second depth, a third contour associated with the first object;
determine, based on the depth data associated with the first image at a second depth, a fourth contour associated with the at least one other object, wherein the third contour and the fourth contour are determined within the single contour;]
determine candidate identifiers for the tracked first object, wherein the candidate identifiers comprise a subset of identifiers of all tracked objects (Siddiquie (col 2 lines 35-53) teaches comparing the embedded vector of features of the object being tracked that may or may not need to be re-identified with other embedded vectors of other tracked agents/objects. Therefore, the subset that is the embedded vectors of the identifier of the “first object” are compared to other identifier embedded vectors of other tracklet tracked objects/agents);
determine, based at least in part on the received first image and the third contour, an updated identifier for the tracked first object, wherein the updated identifier is one of the candidate identifiers (Siddiquie, column 4 line 54 through column 5 line 45, the foreground and background segmentations (plurality of contours, including at least a 1st-third contour) are used in re-identification of the agents in the video, where the embedding vectors are the updated identifiers);
[determine, from the first image, that a portion of the first object has crossed a second predefined zone, wherein the second predefined zone defines a virtual curtain in front of the rack;
and in response to determining that the portion of the first object has crossed the second predefined zone, assign the updated identifier to the tracked first.]
Siddiquie does not teach; determine, based on depth data associated with the first image at the first depth, that the first contour associated with the first object has merged with the second contour associated with the at least one other object to form a single contour;
determine that re-identification of the tracked first object is needed based at least in part upon the determination that the first contour merged with the second contour to form the single contour;
in response to determining that re-identification of the tracked first object is needed;
determine, based on the depth data associated with the first image at a second depth, a third contour associated with the first object;
determine, based on the depth data associated with the first image at a second depth, a fourth contour associated with the at least one other object, wherein the third contour and the fourth contour are determined within the single contour;
determine, from the first image, that a portion of the first object has crossed a second predefined zone, wherein the second predefined zone defines a virtual curtain in front of the rack;]
and in response to determining that the portion of the first object has crossed the second predefined zone, assign the updated identifier to the tracked first object.
However, in the same field of endeavor Zhou teaches;
determine, based on depth data associated with the first image at the first depth, that the first contour associated with the first object has merged with the second contour associated with the at least one other object to form a single contour (Zhou, Column 10, lines, 59-67, and column 11 lines 1-5, trackers can determined whether a merge event has taken place, Column 11, lines 53-58, the system detects blobs to be tracked and identified in video frames, Column 17, Lines 10-15 the blob processing engine can perform processing to further process the blobs, including generating bounding boxes and tracking the blobs, Column 17, lines 29-44, the system may merge some connected components, to combine blobs, to create one large blob, to decrease the chance that small blobs all belong to the same object, further column 11 line 43- column 12 line 15 denotes that the blobs are processed such as to detect whether they are background or foreground blobs, which the examiner is interpreting as depth information);
determine that re-identification of the tracked first object is needed based at least in part upon the determination that the first contour merged with the second contour to form the single contour (Zhou, Column 1, lines 58-67, the blob represents and object or person to be tracked in a video and re-identified between frames. Object is being tracked, blob is being interpreted here as being functionally equivalent to the contour, as described on page 73, Lines 20-30, in the Auto Exclusion Zones sections of Applicant’s specification. In this section applicant define a contour as an edge associated with a person or target object therefore it is functionally equivalent to the detections of Zhou. Column 11, lines 53-58, the system detects blobs to be tracked and identified in video frames, Column 17, Lines 10-15 the blob processing engine can perform processing to further process the blobs, including generating bounding boxes and tracking the blobs, Column 17, lines 29-44, the system may merge some connected components, to combine blobs, to create one large blob, to decrease the chance that small blobs all belong to the same object);
in response to determining that re-identification of the tracked first object is needed (Zhou, Column 1, lines 58-67, the blob represents and object or person to be tracked in a video and re-identified between frames. Object is being tracked, blob is being interpreted here as being functionally equivalent to the contour, as described on page 73, Lines 20-30, in the Auto Exclusion Zones sections of Applicant’s specification. In this section applicant define a contour as an edge associated with a person or target object therefore it is functionally equivalent to the detections of Zhou. Column 11, lines 53-58, the system detects blobs to be tracked and identified in video frames, Column 17, Lines 10-15 the blob processing engine can perform processing to further process the blobs, including generating bounding boxes and tracking the blobs, Column 17, lines 29-44, the system may merge some connected components, to combine blobs, to create one large blob, to decrease the chance that small blobs all belong to the same object);
PNG
media_image1.png
196
448
media_image1.png
Greyscale
(Zhou, column 1, emphasis added)
PNG
media_image2.png
154
424
media_image2.png
Greyscale
(Zhou, Column 11, emphasis added)
PNG
media_image3.png
258
658
media_image3.png
Greyscale
(Zhou, Column 17, emphasis added)
PNG
media_image4.png
326
450
media_image4.png
Greyscale
(Zhou, Column 17, lines 29-44, emphasis added)
The combination of Siddiquie and Zhou would have been obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. The motivation for the combination lies in that the addition of the ability to merge “blobs” or contours to aide in the identification of objects or persons on video would have improved the system of Siddiquie in that it would reduce mis-identification of persons on the video. (Zhou Column 1, lines 58-67, Column 11, lines 53-58, Column 1 lines 67- Col 2 Lines 1-3, Column 17, lines 29-44)
Neither Siddiquie or Zhou teaches;
determine, based on the depth data associated with the first image at a second depth, a third contour associated with the first object;
determine, based on the depth data associated with the first image at a second depth, a fourth contour associated with the at least one other object, wherein the third contour and the fourth contour are determined within the single contour;
determine, from the first image, that a portion of the first object has crossed a second predefined zone, wherein the second predefined zone defines a virtual curtain in front of the rack;]
and in response to determining that the portion of the first object has crossed the second predefined zone, assign the updated identifier to the tracked first object.
However, in the same field of endeavor of motion detection, Zhang teaches;
determine, based on the depth data associated with the first image at a second depth, a third contour associated with the first object (Zhang, [0048] the system detects multiple blobs (contours) corresponding to multiple objects tracked, a blob is a group of pixels corresponding to an object, or a mask of an object in the image which can be used to distinguish targets in an image from one another, [0061] and [0062 further note these blobs are defined by boundary lines on the object, The examine notes that in the applicants specification, page Auto-exclusion zones section, the applicant defines the contours as a curve associated with the edge representation of a person or object, as well as these contours having pixel coordinates, therefore the blobs of Zhang would be analogous to this as described in [0048], [0052] depth data may be extracted from each blob (contour), where one or more blobs (contour) corresponds to one or more real world objects, a single blob (contour) may be determined to correspond to two different objects, further a blob (contour) may contain only part of an object, so an additional blob may be collected to correspond to that target, the examiner is interpreting this as two blobs/contours may be collected for a single object (first contour and third contour) and depth information may be collected accordingly for each blob/contour (at least a first and second depth));
determine, based on the depth data associated with the first image at a second depth, a fourth contour associated with the at least one other object (Zhang, [0048] the system detects multiple blobs (contours) corresponding to multiple objects tracked, a blob is a group of pixels corresponding to an object, or a mask of an object in the image which can be used to distinguish targets in an image from one another, [0061] and [0062 further note these blobs are defined by boundary lines on the object, The examine notes that in the applicants specification, page Auto-exclusion zones section, the applicant defines the contours as a curve associated with the edge representation of a person or object, as well as these contours having pixel coordinates, therefore the blobs of Zhang would be analogous to this as described in [0048], [0052] depth data may be extracted from each blob (contour), where one or more blobs (contour) corresponds to one or more real world objects, a single blob (contour) may be determined to correspond to two different objects, further a blob (contour) may contain only part of an object, so an additional blob may be collected to correspond to that target, the examiner is interpreting this as two blobs/contours may be collected for a single object (first contour and third contour) and depth information may be collected accordingly for each blob/contour (at least a first and second depth), further given that a plurality of blobs are detected for a plurality of objects, multiple objects may be within the same blob, in which the system would separate those into separate contours later, included within the same blob, but corresponding to separate objects, indicating at least a second and a fourth contour/blob and corresponding depth information),
wherein the third contour and the fourth contour are determined within the single contour (Zhang, [0052] depth data may be extracted from each blob (contour), where one or more blobs (contour) corresponds to one or more real world objects, a single blob (contour) may be determined to correspond to two different objects, further a blob (contour) may contain only part of an object, so an additional blob may be collected to correspond to that target, the examiner is interpreting this as two blobs/contours may be collected for a single object (first contour and third contour) and depth information may be collected accordingly for each blob/contour (at least a first and second depth), further given that a plurality of blobs are detected for a plurality of objects, multiple objects may be within the same blob, in which the system would separate those into separate contours later, included within the same blob, but corresponding to separate objects, indicating at least a second and a fourth contour/blob and corresponding depth information, since multiple objects can exist within a single blob, and then be separated into individual contours, the examiner is interpreting this as the contours may exist within one contour/curve);
The combination of Siddiquie, Zhou and Zhang would be obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. The combination of Siddiquie and Zhou fails to teach the determination of a third and fourth contour with depth information belonging to a single contour. In the same field of endeavor of object tracking and re-identification, Zhang teaches this deficiency. The motivation to add this feature of Zhang lies in that the inclusion of more comprehensive depth information to the detection capacities of Siddiquie and Zhou allows for features of the person being tracked, such as height or size to be more accurately detected, as well as more accurate 3D position detection. (Zhang [0042]- [0045])
Further, Siddiquie, Zhou and Zhang fail to teach;
determine, from the first image, that a portion of the first object has crossed a second predefined zone, wherein the second predefined zone defines a virtual curtain in front of the rack;
and in response to determining that the portion of the first object has crossed the second predefined zone, assign the updated identifier to the tracked first object.
However, in the same field of endeavor, Fisher teaches;
determine, from the first image, that a portion of the first object has crossed a second predefined zone, wherein the second predefined zone defines a virtual curtain in front of the rack (Fisher, column 23 lines 1-17, the cameras use a time series of images (at least a first image) to determine if an object is picked up from a shelf and removed, where the distance of the person’s hand from the shelf is a known distance/area, this is being interpreted as being analogous to a second predefined zone in front of a shelf, column 23 line 49- 58 the shelf (zone 1) is mapped and the areas in front of the shelves and the rest of the store are mapped as well (at least a second zone in front of the shelf), whether or not items have been removed from the shelves is tracked, therefore it would be determined if an item exits the shelf (zone 1) and enters the region in front of the shelf (zone 2), Figure 14 and pages 4 and 5 of applicant’s specification define the virtual curtain as a zone in front of the rack which the objects pass through to determine if they have been removed from a rack or shelf. );
PNG
media_image5.png
170
320
media_image5.png
Greyscale
(Fisher, column 23, emphasis added)
and in response to determining that the portion of the first object has crossed the second predefined zone, assign the updated identifier to the tracked first object (Fisher, column 23 lines 1-17, the cameras use a time series of images (at least a first image) to determine if an object is picked up from a shelf and removed, where the distance of the person’s hand from the shelf is a known distance/area, this is being interpreted as being analogous to a second predefined zone in front of a shelf, column 23 line 49- 58 the shelf (zone 1) is mapped and the areas in front of the shelves and the rest of the store are mapped as well (at least a second zone in front of the shelf), whether or not items have been removed from the shelves is tracked, therefore it would be determined if an item exits the shelf (zone 1) and enters the region in front of the shelf (zone 2), once an object is determined as being removed from the shelf (entering zone 2) its index number is updated (updated identifier is assigned)).
The combination of Siddiquie, Zhou, Zhang and Fisher would be obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. The motivation for the addition of the determination that the object enters a second zone in front of the shelf as taught by Fisher is that it allows for accurate tracking of inventory items which have been removed from the shelf by a person for ease of virtual payment. (Fisher, column 23)
Regarding claim 2 the combination of Siddiquie, Zhou, Zhang and Fisher teach; The system of Claim 1, wherein the processor of the tracking subsystem is further configured to (Siddiquie Column 4 lines 55-60, A processor unit is configured to operate capabilities of the system):
determine, based on the first image, a first feature value associated with observable characteristics of the tracked first object (Siddiquie (col 4 line 52 through col 5 line 20) discloses assigning to each agent (each agent corresponding to a worker/user/person/object) a feature value that is an embedding vector (col 5 lines 1-15, embedding vector EV110 corresponding to 1st agent, EV120 to 2nd agent, EV130 to 3rd agent). Siddique (col 5 lines 35-45) has further disclosed that these embedding vectors describe observable object attributes including but not limited to color, size, shape, texture. The resultant embedding vectors assigned to each agent (180) or tracklet to produce feature sets (188) (col 6 lines 20-35)),
compare the first feature value to a set of predetermined feature values associated with the candidate identifiers determined for the tracked first object (Siddiquie (col 6 lines 30-43) discloses the set of new and old embedded vector type feature values are compared to determine similarity with an associated agent, hence embedded vector of image data for an agent/object is compared to one or more candidate identifier “embedded vectors” for the first and so on tracked agent/person/object of Siddiquie);
based on results of the comparison, determine the updated identifier for the tracked first object, wherein the updated identifier is the predetermined feature value from the set of predetermined feature values with a value that is within a threshold range of the first feature value (Siddiquie, (col 2 lines 45-67) discloses selecting a “first object” candidate that is one of the multiple objects (agents) in the image data who has a “candidate identifier” most similar to the previous identifier type embedded vector of the Agent_1/Agent_A. New embedded vector data of the identifier for the Agent_1/”first object” is determined based on the similarity determination of the candidate identifiers (col 2 lines 45-67). Siddiquie (col 6 lines 25-40) has disclosed replacing or further including the identifier/” embedding vector”/features calculated for the agent/object).
Regarding claim 3 the combination of Siddiquie, Zhou, Zhang and Fisher teach;
The system of Claim 2, wherein: the first feature value comprises a first data vector associated with the observable characteristics of the tracked first object (Siddiquie (col 4 line 52 through col 5 line 20) discloses assigning to each agent 180-1 - 180-4 (each agent corresponding to a person/object) a feature value that is an embedding vector (col 5 lines 1-15, embedding vector EV110 corresponding to 1st agent, EV120 to 2nd agent, EV130 to 3rd agent, and so on). Siddique (col 5 lines 35-45) has further disclosed that these embedding vectors describe observable object attributes including but not limited to color, size, shape, texture. The resultant embedding vectors assigned to each agent 180 or tracklet to produce feature sets 188 (col 6 lines 20-35). The embedding vector for an agent comprising at least some “first” value else no value would be assigned to said corresponding agent);
each of the predetermined feature values from the set of predetermined feature values comprises a corresponding predetermined data vector (Siddiquie, Col 4, line 52 – col 5 line 20 teaches at least one embedding vector associated with each respective agent/object);
and the processor of the tracking subsystem is further configured to (Siddiquie, Column 3, Lines 55-66, the system has a processor unit (134)):
compare the first feature value to each of the predetermined feature values from the set of predetermined feature values associated with the candidate identifiers by calculating a similarity value between the first data vector and each of the predetermined data vectors (Siddiquie (col 2 lines 45-67) discloses selecting a “first object” that is one of the plurality objects in the image data who has a “candidate identifier” most similar to the previous identifier type embedded vector of the Agent_1/Agent_A. New embedded vector data of the identifier for the Agent_1/”first object” is determined based on the similarity determination of the candidate identifiers (col 2 lines 45-67). Siddiquie (col 6 lines 25-40) has disclosed replacing or further including the identifier/”embedding vector”/features calculated for the agent/object);
and determine the updated identifier as a particular candidate identifier that corresponds to the similarity value that is nearest one (Siddiquie (col 6 lines 20-43) discloses updating weighting and inclusion of new embedding vectors “identifiers” based on a similarity comparison determination of closeness of match of said feature type embedding vectors).
Regarding claim 4 the combination of Siddiquie, Zhou, Zhang and Fisher teach;
The system of Claim 3, wherein the processor (Siddiquie, Column 3, Lines 55-66, the system has a processor unit (134)) of the tracking subsystem is further configured to determine that each of the similarity values is less than a threshold similarly value and, in response (Siddiquie, Column 19 lines 64-column 20 lines 1-9, The system generates a similarity score for the agent (object) identified, and confidence score, features are compared, the confidence score is then determined to exceed a threshold to determine the agent, Examiner is interpreting this as the system has the ability to determine if a similarity threshold exceeds or falls below a certain level):
determine a second feature value for each of the one or more other objects, wherein each second feature value comprises a second data vector (Siddiquie (col 4 line 52 through col 5 line 20) discloses assigning to each of agent (each agent corresponding to a worker/user/person/object) a feature value that is an embedding vector (col 5 lines 1-15, embedding vector EV110 corresponding to 1st agent, EV120 to 2nd agent, EV130 to 3rd agent, and so on). Siddique (col 5 lines 35-45) has further disclosed that these embedding vectors describe observable object attributes including but not limited to color, size, shape, texture, The resultant embedding vectors assigned to each agent 180 or tracklet to produce feature sets 188 (col 6 lines 20-35). Multiple embedding vectors, which would include second, third, and so on feature values are associated with the one or more agents (col 5 lines 50-67, col 6 lines 1-10, col 6 lines 20-43, col 6 lines 50-65, col 7 lines 1-31));
determine a second similarity value between each of the second data vectors and each of the predetermined feature values (Siddiquie (col 4 line 52 through col 5 line 20) discloses assigning to each of agent (each agent corresponding to a worker/user/person/object) a feature value that is an embedding vector (col 5 lines 1-15, embedding vector EV110 corresponding to 1st agent, EV120 to 2nd agent, EV130 to 3rd agent, and so on). Siddique (col 5 lines 35-45) further discloses that these embedding vectors describe observable object attributes including but not limited to color, size, shape, texture. The resultant embedding vectors assigned to each agent 180 or tracklet to produce feature sets 188 (col 6 lines 20-35). Multiple embedding vectors, which would include a second and third feature values are associated with the one or more agents (col 5 lines 50-67, col 6 lines 1-10, col 6 lines 20-43, col 6 lines 50-65, col 7 lines 1-31));
and determine second updated identifiers for the tracked first object and each of the one or more other objects, based on the first and second similarity values (Siddiquie discloses (col 4 line 52 through col 5 line 20, col 5 lines 35-45, col 5 lines 50-67, col 6 lines 1-10, col 6 lines 20-43, col 6 lines 50-65, col 7 lines 1-31 as cited above) performing a plurality of similarity determinations (Including at least a first and second) for first and second embedded vectors for first second and so on agents/objects in order to determine and update a set of feature value/”embedded vectors”/identifiers for the plurality of agents).
Regarding claim 5 the combination of Siddiquie, Zhou, Zhang and Fisher teaches; The system of Claim 1, wherein the updated identifier is determined based on a portion of the first image, the portion corresponding to a predefined field-of-view comprising a central sub-region of a full field-of-view addressed by the first sensor (Siddiquie (col 3 lines 35-65) discloses extracting a corresponding central/foreground region of the image data out of the full view of the captured video image data, and segmenting said foreground/central portion for further processing of the foreground/central object regions, (col 6, Lines 25-30) In some implementations the field of view may be processed in the feature set for that agent, which is part of the object/agents identifier).
Regarding claim 6 the combination of Siddiquie, Zhou, Zhang and Fisher teach; (Original) The system of Claim 1, wherein the tracked first object is a first person (Siddiquie (col 8 lines 20-35) discloses using a plurality of digital imaging devices to capture overlapping fields of view of data in a facility (col 2 lines 7-37). The image devices suppling image data to track users/objects/workers/agents in a facility using the disclosed computing device of Fig 4 (“tracklet” of the one or more agents – col 2 lines 7-37), the tracklet showing the tracked object is a person).
Regarding claim 7 the combination of Siddiquie, Zhou, Zhang and Fisher teaches; The system of Claim 2, wherein the processor of the tracking subsystem is further configured to, prior to determining that re-identification of the tracked first object is needed, periodically determine updated predetermined feature values associated with the candidate identifiers (Siddiquie (col 6 lines 20-30) has disclosed periodically updating the embedded vectors (candidate identifiers) that are the feature sets identifying the objects/agent being tracking the image processing system).
Regarding claim 8 the combination of Siddiquie, Zhou, Zhang and Fisher teach; A method comprising:
receiving images generated by a plurality of sensors (Siddiquie, Column 3, Lines 55-66, the system has a processor unit (134) which may receive images);
tracking a first object and one or more other objects in a space using at least a portion of the images generated by the plurality of sensors (Siddiquie, Column 3, Lines 42-53, Locations of agents may be tracked via a “tracklet” which supplies the position of multiple agents via positions detection in image frames),
wherein the first object is associated with a first contour at a first depth and at least one of the other objects in the space is associated with a second contour at the first depth (Siddiquie, column 4 lines 38-52, images collected can be depth images, whereas per column 3 line 54 through column 4 line 37 the objects may be distinguished using background and foreground segmentations and other background vs foreground identification methods to distinguish objects to be tracked vs not to be tracked, the Examiner is interpreting this as depth information associated with the contour), wherein each of the first contour and the second contour comprises a curve associated with an edge of a representation of the respective first object or the at least one other object (Siddiquie (col 8 lines 20-35) discloses using a plurality of digital imaging devices to capture overlapping fields of view of data in a facility (col 2 lines 7-37). The image devices supplying image data to track users in a facility using the disclosed computing device of at least Fig 4 (“tracklet” of the one or more agents – col 2 lines 7-37). Siddique further disclosed (Fig 1H and Fig 1i items 180-1, 180-2, 180-3, 180-4, depicted as tracked rectangular blocks) two-person type objects being tracked, wherein each object is tracked based on contours instead of the blocks depicted in the figures (col 10 lines 19-40). The objects having a same or at least overlapping position (see Fig 1H and Fig 1i items 180-1 and overlapping/merged item 180-2, depicted as tracked rectangular blocks), wherein the determination that a same or overlapping object region corresponds to a request for object re-identification (a low confidence condition is initiated for proximate/merged/overlapping objects – col 19 lines 40-53, identified objects/agents of low confidence used for re-identification: col 19 lines 50-67, Column 10, lines 20-51, the system generates contours, shapes or outlines of the objects being tracked);
receiving, from a first sensor of the plurality of sensors, a first image of the tracked first object (Siddiquie, column 2, lines 5-36, the system has a plurality of imaging devices set to capture a plurality of images of objects and agents, indicating at least a first and a second tracked object);
determining, based on depth data associated with the first image at the first depth, that the first contour associated with the first object has merged with the second contour associated with the at least one other object to form a single contour (Zhou, Column 10, lines, 59-67, and column 11 lines 1-5, trackers can determined whether a merge event has taken place, Column 11, lines 53-58, the system detects blobs to be tracked and identified in video frames, Column 17, Lines 10-15 the blob processing engine can perform processing to further process the blobs, including generating bounding boxes and tracking the blobs, Column 17, lines 29-44, the system may merge some connected components, to combine blobs, to create one large blob, to decrease the chance that small blobs all belong to the same object, further column 11 line 43- column 12 line 15 denotes that the blobs are processed such as to detect whether they are background or foreground blobs, which the examiner is interpreting as depth information);
determine, from the first image, that the first object and the at least one other object have crossed a first predefined zone, wherein the first predefined zone identifies an area associated with a rack (Siddiquie, column 3 line 54- column 4 lines 36 teaches that objects may be tracked using bounding boxes (zones), and in column 10 line 20-51 the objects may be labeled and tracked when in motion, column 11 lines 10-32 and column 12 lines 10-33, items in an inventory at a store may be tracked, and the moved position and distance moved of the item are tracked, which indicates that the item is moving between “zones” and is tracked/detected. Figures 5a and 5b show this as an overhead where all the items are in a grid system, or “zones” and are tracked);
determining that re-identification of the tracked first object is needed based at least in part upon the determination that the first contour merged with the second contour to form the single contour (Zhou, Column 1, lines 58-67, the blob represents and object or person to be tracked in a video and re-identified between frames. Object is being tracked, blob is being interpreted here as being functionally equivalent to the contour, as described on page 73, Lines 20-30, in the Auto Exclusion Zones sections of Applicant’s specification. In this section applicant define a contour as an edge associated with a person or target object therefore it is functionally equivalent to the detections of Zhou. Column 11, lines 53-58, the system detects blobs to be tracked and identified in video frames, Column 17, Lines 10-15 the blob processing engine can perform processing to further process the blobs, including generating bounding boxes and tracking the blobs, Column 17, lines 29-44, the system may merge some connected components, to combine blobs, to create one large blob, to decrease the chance that small blobs all belong to the same object);
in response to determining that re-identification of the tracked first object is needed (Zhou, Column 1, lines 58-67, the blob represents and object or person to be tracked in a video and re-identified between frames. Object is being tracked, blob is being interpreted here as being functionally equivalent to the contour, as described on page 73, Lines 20-30, in the Auto Exclusion Zones sections of Applicant’s specification. In this section applicant define a contour as an edge associated with a person or target object therefore it is functionally equivalent to the detections of Zhou. Column 11, lines 53-58, the system detects blobs to be tracked and identified in video frames, Column 17, Lines 10-15 the blob processing engine can perform processing to further process the blobs, including generating bounding boxes and tracking the blobs, Column 17, lines 29-44, the system may merge some connected components, to combine blobs, to create one large blob, to decrease the chance that small blobs all belong to the same object);
determining, based on the depth data associated with the first image at a second depth, a third contour associated with the first object ((Zhang, [0048] the system detects multiple blobs (contours) corresponding to multiple objects tracked, a blob is a group of pixels corresponding to an object, or a mask of an object in the image which can be used to distinguish targets in an image from one another, [0061] and [0062 further note these blobs are defined by boundary lines on the object, The examine notes that in the applicants specification, page Auto-exclusion zones section, the applicant defines the contours as a curve associated with the edge representation of a person or object, as well as these contours having pixel coordinates, therefore the blobs of Zhang would be analogous to this as described in [0048], [0052] depth data may be extracted from each blob (contour), where one or more blobs (contour) corresponds to one or more real world objects, a single blob (contour) may be determined to correspond to two different objects, further a blob (contour) may contain only part of an object, so an additional blob may be collected to correspond to that target, the examiner is interpreting this as two blobs/contours may be collected for a single object (first contour and third contour) and depth information may be collected accordingly for each blob/contour (at least a first and second depth));
determining, based on the depth data associated with the first image at a second depth, a fourth contour associated with the at least one other object (Zhang, [0048] the system detects multiple blobs (contours) corresponding to multiple objects tracked, a blob is a group of pixels corresponding to an object, or a mask of an object in the image which can be used to distinguish targets in an image from one another, [0061] and [0062 further note these blobs are defined by boundary lines on the object, The examine notes that in the applicants specification, page Auto-exclusion zones section, the applicant defines the contours as a curve associated with the edge representation of a person or object, as well as these contours having pixel coordinates, therefore the blobs of Zhang would be analogous to this as described in [0048], [0052] depth data may be extracted from each blob (contour), where one or more blobs (contour) corresponds to one or more real world objects, a single blob (contour) may be determined to correspond to two different objects, further a blob (contour) may contain only part of an object, so an additional blob may be collected to correspond to that target, the examiner is interpreting this as two blobs/contours may be collected for a single object (first contour and third contour) and depth information may be collected accordingly for each blob/contour (at least a first and second depth), further given that a plurality of blobs are detected for a plurality of objects, multiple objects may be within the same blob, in which the system would separate those into separate contours later, included within the same blob, but corresponding to separate objects, indicating at least a second and a fourth contour/blob and corresponding depth information),
wherein the third contour and the fourth contour are determined within the single contour (Zhang, [0052] depth data may be extracted from each blob (contour), where one or more blobs (contour) corresponds to one or more real world objects, a single blob (contour) may be determined to correspond to two different objects, further a blob (contour) may contain only part of an object, so an additional blob may be collected to correspond to that target, the examiner is interpreting this as two blobs/contours may be collected for a single object (first contour and third contour) and depth information may be collected accordingly for each blob/contour (at least a first and second depth), further given that a plurality of blobs are detected for a plurality of objects, multiple objects may be within the same blob, in which the system would separate those into separate contours later, included within the same blob, but corresponding to separate objects, indicating at least a second and a fourth contour/blob and corresponding depth information, since multiple objects can exist within a single blob, and then be separated into individual contours, the examiner is interpreting this as the contours may exist within one contour/curve);
determining candidate identifiers for the tracked first object, wherein the candidate identifiers comprise a subset of identifiers of all tracked objects (Siddiquie (col 2 lines 35-53) teaches comparing the embedded vector of features of the object being tracked that may or may not need to be re-identified with other embedded vectors of other tracked agents/objects. Therefore, the subset that is the embedded vectors of the identifier of the “first object” are compared to other identifier embedded vectors of other tracklet tracked objects/agents);
determine, from the first image, that a portion of the first object has crossed a second predefined zone, wherein the second predefined zone defines a virtual curtain in front of the rack (Fisher, column 23 lines 1-17, the cameras use a time series of images (at least a first image) to determine if an object is picked up from a shelf and removed, where the distance of the person’s hand from the shelf is a known distance/area, this is being interpreted as being analogous to a second predefined zone in front of a shelf, column 23 line 49- 58 the shelf (zone 1) is mapped and the areas in front of the shelves and the rest of the store are mapped as well (at least a second zone in front of the shelf), whether or not items have been removed from the shelves is tracked, therefore it would be determined if an item exits the shelf (zone 1) and enters the region in front of the shelf (zone 2), Figure 14 and pages 4 and 5 of applicant’s specification define the virtual curtain as a zone in front of the rack which the objects pass through to determine if they have been removed from a rack or shelf. );
PNG
media_image5.png
170
320
media_image5.png
Greyscale
(Fisher, column 23, emphasis added)
and in response to determining that the portion of the first object has crossed the second predefined zone, assign the updated identifier to the tracked first object (Fisher, column 23 lines 1-17, the cameras use a time series of images (at least a first image) to determine if an object is picked up from a shelf and removed, where the distance of the person’s hand from the shelf is a known distance/area, this is being interpreted as being analogous to a second predefined zone in front of a shelf, column 23 line 49- 58 the shelf (zone 1) is mapped and the areas in front of the shelves and the rest of the store are mapped as well (at least a second zone in front of the shelf), whether or not items have been removed from the shelves is tracked, therefore it would be determined if an item exits the shelf (zone 1) and enters the region in front of the shelf (zone 2), once an object is determined as being removed from the shelf (entering zone 2) its index number is updated (updated identifier is assigned)).
The combination of Siddiquie, Zhou, Zhang and Fisher would be obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. The motivation for the combination lies in that the addition of the ability to merge “blobs” or contours to aide in the identification of objects or persons on video would have improved the system of Siddiquie in that it would reduce mis-identification of persons on the video (Zhou Column 1, lines 58-67, Column 11, lines 53-58, Column 1 lines 67- Col 2 Lines 1-3, Column 17, lines 29-44). Further, the combination of Siddiquie, Zhou and Zhang would be obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. The combination of Siddiquie and Zhou fails to teach the determination of a third and fourth contour with depth information belonging to a single contour. In the same field of endeavor of object tracking and re-identification, Zhang teaches this deficiency. The motivation to add this feature of Zhang lies in that the inclusion of more comprehensive depth information to the detection capacities of Siddiquie and Zhou allows for features of the person being tracked, such as height or size to be more accurately detected, as well as more accurate 3D position detection (Zhang [0042]- [0045]. The motivation for the addition of the determination that the object enters a second zone in front of the shelf as taught by Fisher is that it allows for accurate tracking of inventory items which have been removed from the shelf by a person for ease of virtual payment. (Fisher, column 23)
Regarding claim 9 the combination of Siddiquie, Zhou, Zhang and Fisher teach; The method of Claim 8, further comprising:
determining, based on the first image, a first feature value associated with observable characteristics of the tracked first object (Siddiquie (col 4 line 52 through col 5 line 20) discloses assigning to each agent (each agent corresponding to a worker/user/person/object) a feature value that is an embedding vector (col 5 lines 1-15, embedding vector EV110 corresponding to 1st agent, EV120 to 2nd agent, EV130 to 3rd agent). Siddiquie (col 5 lines 35-45) has further disclosed that these embedding vectors describe observable object attributes including but not limited to color, size, shape, texture. The resultant embedding vectors assigned to each agent (180) or tracklet to produce feature sets (188) (col 6 lines 20-35));
comparing the first feature value to a set of predetermined feature values associated with the candidate identifiers determined for the tracked first object (Siddiquie (col 6 lines 30-43) discloses the set of new and old embedded vector type feature values are compared to determine similarity with an associated agent, hence embedded vector of image data for an agent/object is compared to one or more candidate identifier “embedded vectors” for the first and so on tracked agent/person/object of Siddiquie);
based on results of the comparison, determining the updated identifier for the tracked first object, wherein the updated identifier is the predetermined feature value from the set of predetermined feature values with a value that is within a threshold range of the first feature value (Siddiquie, (col 2 lines 45-67) discloses selecting a “first object” candidate that is one of the multiple objects (agents) in the image data who has a “candidate identifier” most similar to the previous identifier type embedded vector of the Agent_1/Agent_A. New embedded vector data of the identifier for the Agent_1/”first object” is determined based on the similarity determination of the candidate identifiers (col 2 lines 45-67). Siddiquie (col 6 lines 25-40) has disclosed replacing or further including the identifier/”embedding vector”/features calculated for the agent/object).
Regarding claim 10 the combination of Siddiquie, Zhou, Zhang and Fisher teach;
The method of Claim 9, wherein: the first feature value comprises a first data vector associated with the observable characteristics of the tracked first object (Siddiquie (col 4 line 52 through col 5 line 20) discloses assigning to each agent 180-1 - 180-4 (each agent corresponding to a person/object) a feature value that is an embedding vector (col 5 lines 1-15, embedding vector EV110 corresponding to 1st agent, EV120 to 2nd agent, EV130 to 3rd agent, and so on). Siddique (col 5 lines 35-45) has further disclosed that these embedding vectors describe observable object attributes including but not limited to color, size, shape, texture. The resultant embedding vectors assigned to each agent 180 or tracklet to produce feature sets 188 (col 6 lines 20-35). The embedding vector for an agent comprising at least some “first” value else no value would be assigned to said corresponding agent);
each of the predetermined feature values from the set of predetermined feature values comprises a corresponding predetermined data vector (Siddiquie, Col 4, line 52 – col 5 line 20 teaches at least one embedding vector associated with each respective agent/object);
and the method further comprises: comparing the first feature value to each of the predetermined feature values from the set of predetermined feature values associated with the candidate identifiers by calculating a similarity value between the first data vector and each of the predetermined data vectors (Siddiquie (col 2 lines 45-67) discloses selecting a “first object” that is one of the plurality objects in the image data who has a “candidate identifier” most similar to the previous identifier type embedded vector of the Agent_1/Agent_A. New embedded vector data of the identifier for the Agent_1/”first object” is determined based on the similarity determination of the candidate identifiers (col 2 lines 45-67). Siddiquie (col 6 lines 25-40) has disclosed replacing or further including the identifier/”embedding vector”/features calculated for the agent/object);
and determining the updated identifier as a particular candidate identifier that corresponds to the similarity value that is nearest one (Siddiquie (col 6 lines 20-43) discloses updating weighting and inclusion of new embedding vectors “identifiers” based on a similarity comparison determination of closeness of match of said feature type embedding vectors).
Regarding claim 11 the combination of Siddiquie, Zhou, Zhang and Fisher teach;
The method of Claim 10, further comprising determining that each of the similarity values is less than a threshold similarly value and, in response (Siddiquie, Column 19 lines 64-column 20 lines 1-9, The system generates a similarity score for the agent (object) identified, and confidence score, features are compared, the confidence score is then determined to exceed a threshold to determine the agent, Examiner is interpreting this as the system has the ability to determine if a similarity threshold exceeds or falls below a certain level):
determining a second feature value for each of the one or more other objects, wherein each second feature value comprises a second data vector (Siddiquie (col 4 line 52 through col 5 line 20) discloses assigning to each of agent (each agent corresponding to a worker/user/person/object) a feature value that is an embedding vector (col 5 lines 1-15, embedding vector EV110 corresponding to 1st agent, EV120 to 2nd agent, EV130 to 3rd agent, and so on). Siddique (col 5 lines 35-45) has further disclosed that these embedding vectors describe observable object attributes including but not limited to color, size, shape, texture, The resultant embedding vectors assigned to each agent 180 or tracklet to produce feature sets 188 (col 6 lines 20-35). Multiple embedding vectors, which would include second, third, and so on feature values are associated with the one or more agents (col 5 lines 50-67, col 6 lines 1-10, col 6 lines 20-43, col 6 lines 50-65, col 7 lines 1-31));
determining a second similarity value between each of the second data vectors and each of the predetermined feature values (Siddiquie (col 4 line 52 through col 5 line 20) discloses assigning to each of agent (each agent corresponding to a worker/user/person/object) a feature value that is an embedding vector (col 5 lines 1-15, embedding vector EV110 corresponding to 1st agent, EV120 to 2nd agent, EV130 to 3rd agent, and so on). Siddique (col 5 lines 35-45) further discloses that these embedding vectors describe observable object attributes including but not limited to color, size, shape, texture. The resultant embedding vectors assigned to each agent 180 or tracklet to produce feature sets 188 (col 6 lines 20-35). Multiple embedding vectors, which would include a second and third feature values are associated with the one or more agents (col 5 lines 50-67, col 6 lines 1-10, col 6 lines 20-43, col 6 lines 50-65, col 7 lines 1-31));
and determining second updated identifiers for the tracked first object and each of the one or more other objects, based on the first and second similarity values (Siddiquie discloses (col 4 line 52 through col 5 line 20, col 5 lines 35-45, col 5 lines 50-67, col 6 lines 1-10, col 6 lines 20-43, col 6 lines 50-65, col 7 lines 1-31 as cited above) performing a plurality of similarity determinations (Including at least a first and second) for first and second embedded vectors for first second and so on agents/objects in order to determine and update a set of feature value/”embedded vectors”/identifiers for the plurality of agents).
Regarding claim 12 the combination of Siddiquie, Zhou, Zhang and Fisher teaches; The method of Claim 8, further comprising determining the updated identifier based on a portion of the first image, the portion corresponding to a predefined field- of-view comprising a central sub-region of a full field-of-view addressed by the first sensor (Siddiquie (col 3 lines 35-65) discloses extracting a corresponding central/foreground region of the image data out of the full view of the captured video image data, and segmenting said foreground/central portion for further processing of the foreground/central object regions, (col 6, Lines 25-30) In some implementations the field of view may be processed in the feature set for that agent, which is part of the object/agents identifier).
Regarding claim 13 the combination of Siddiquie, Zhou, Zhang and Fisher teach; The method of Claim 8, wherein the tracked first object is a first person (Siddiquie (col 8 lines 20-35) discloses using a plurality of digital imaging devices to capture overlapping fields of view of data in a facility (col 2 lines 7-37). The image devices suppling image data to track users/objects/workers/agents in a facility using the disclosed computing device of Fig 4 (“tracklet” of the one or more agents – col 2 lines 7-37), the tracklet showing the tracked object is a person).
Regarding claim 14 the combination of Siddiquie, Zhou, Zhang and Fisher teaches; The method of Claim 9, further comprising, prior to determining that re-identification of the tracked first object is needed, periodically determining updated predetermined feature values associated with the candidate identifiers (Siddiquie (col 6 lines 20-30) has disclosed periodically updating the embedded vectors (candidate identifiers) that are the feature sets identifying the objects/agent being tracking the image processing system).
Regarding claim 15 the combination of Siddiquie, Zhou, Zhang and Fisher teaches; A tracking subsystem communicatively coupled to a plurality of sensors (Siddiquie, Column 1, Lines 20-31, the system may include one or more sensors for the capture of image data, Column 11, the system has a re-identification system for tracking agents in the facility), each sensor of the plurality of sensors configured to generate images of at least a portion of a space, the tracking subsystem comprising a processor configured to(Siddiquie Column 2, lines 5-35, one or more external imaging devices to capture an area, Figures 1a-f show the camera’s views for identifying and re-identifying a person in a store):
receive the images generated by the plurality of sensors (Siddiquie, Column 3, Lines 55-66, the system has a processor unit (134) which may receive images);
track a first object and one or more other objects in the space using at least a portion of the images generated by the plurality of sensors (Siddiquie, Column 3, Lines 42-53, Locations of agents may be tracked via a “tracklet” which supplies the position of multiple agents via positions detection in image frames),
wherein the first object is associated with a first contour at a first depth and at least one of the other objects in the space is associated with a second contour at the first depth, wherein each of the first contour and the second contour comprises a curve associated with an edge of a representation of the respective first object or the at least one other object (Siddiquie (col 8 lines 20-35) discloses using a plurality of digital imaging devices to capture overlapping fields of view of data in a facility (col 2 lines 7-37). The image devices supplying image data to track users in a facility using the disclosed computing device of at least Fig 4 (“tracklet” of the one or more agents – col 2 lines 7-37). Siddique further disclosed (Fig 1H and Fig 1i items 180-1, 180-2, 180-3, 180-4, depicted as tracked rectangular blocks) two-person type objects being tracked, wherein each object is tracked based on contours instead of the blocks depicted in the figures (col 10 lines 19-40). The objects having a same or at least overlapping position (see Fig 1H and Fig 1i items 180-1 and overlapping/merged item 180-2, depicted as tracked rectangular blocks), wherein the determination that a same or overlapping object region corresponds to a request for object re-identification (a low confidence condition is initiated for proximate/merged/overlapping objects – col 19 lines 40-53, identified objects/agents of low confidence used for re-identification: col 19 lines 50-67, Column 10, lines 20-51, the system generates contours, shapes or outlines of the objects being tracked);
receive, from a first sensor of the plurality of sensors, a first image of the tracked first object (Siddiquie, column 2, lines 5-36, the system has a plurality of imaging devices set to capture a plurality of images of objects and agents, indicating at least a first and a second tracked object);
determine, based on depth data associated with the first image at the first depth (Siddiquie, column 4 lines 38-52, images collected can be depth images, whereas per column 3 line 54 through column 4 line 37 the objects may be distinguished using background and foreground segmentations and other background vs foreground identification methods to distinguish objects to be tracked vs not to be tracked, the Examiner is interpreting this as depth information associated with the contour), that the first contour associated with the first object has merged with the second contour associated with the at least one other object to form a single contour(Zhou, Column 10, lines, 59-67, and column 11 lines 1-5, trackers can determined whether a merge event has taken place, Column 11, lines 53-58, the system detects blobs to be tracked and identified in video frames, Column 17, Lines 10-15 the blob processing engine can perform processing to further process the blobs, including generating bounding boxes and tracking the blobs, Column 17, lines 29-44, the system may merge some connected components, to combine blobs, to create one large blob, to decrease the chance that small blobs all belong to the same object);
determine, from the first image, that the first object and the at least one other object have crossed a first predefined zone, wherein the first predefined zone identifies an area associated with a rack (Siddiquie, column 3 line 54- column 4 lines 36 teaches that objects may be tracked using bounding boxes (zones), and in column 10 line 20-51 the objects may be labeled and tracked when in motion, column 11 lines 10-32 and column 12 lines 10-33, items in an inventory at a store may be tracked, and the moved position and distance moved of the item are tracked, which indicates that the item is moving between “zones” and is tracked/detected. Figures 5a and 5b show this as an overhead where all the items are in a grid system, or “zones” and are tracked);
determine that re-identification of the tracked first object is needed based at least in part upon the determination that the first contour merged with the second contour to form the single contour (Zhou, Column 1, lines 58-67, the blob represents and object or person to be tracked in a video and re-identified between frames. Object is being tracked, blob is being interpreted here as being functionally equivalent to the contour, as described on page 73, Lines 20-30, in the Auto Exclusion Zones sections of Applicant’s specification. In this section applicant define a contour as an edge associated with a person or target object therefore it is functionally equivalent to the detections of Zhou. Column 11, lines 53-58, the system detects blobs to be tracked and identified in video frames, Column 17, Lines 10-15 the blob processing engine can perform processing to further process the blobs, including generating bounding boxes and tracking the blobs, Column 17, lines 29-44, the system may merge some connected components, to combine blobs, to create one large blob, to decrease the chance that small blobs all belong to the same object)
in response to determining that re-identification of the tracked first object is needed (Zhou, Column 1, lines 58-67, the blob represents and object or person to be tracked in a video and re-identified between frames. Object is being tracked, blob is being interpreted here as being functionally equivalent to the contour, as described on page 73, Lines 20-30, in the Auto Exclusion Zones sections of Applicant’s specification. In this section applicant define a contour as an edge associated with a person or target object therefore it is functionally equivalent to the detections of Zhou. Column 11, lines 53-58, the system detects blobs to be tracked and identified in video frames, Column 17, Lines 10-15 the blob processing engine can perform processing to further process the blobs, including generating bounding boxes and tracking the blobs, Column 17, lines 29-44, the system may merge some connected components, to combine blobs, to create one large blob, to decrease the chance that small blobs all belong to the same object);
determine, based on the depth data associated with the first image at a second depth, a third contour associated with the first object (Zhang, [0048] the system detects multiple blobs (contours) corresponding to multiple objects tracked, a blob is a group of pixels corresponding to an object, or a mask of an object in the image which can be used to distinguish targets in an image from one another, [0061] and [0062 further note these blobs are defined by boundary lines on the object, The examine notes that in the applicants specification, page Auto-exclusion zones section, the applicant defines the contours as a curve associated with the edge representation of a person or object, as well as these contours having pixel coordinates, therefore the blobs of Zhang would be analogous to this as described in [0048], [0052] depth data may be extracted from each blob (contour), where one or more blobs (contour) corresponds to one or more real world objects, a single blob (contour) may be determined to correspond to two different objects, further a blob (contour) may contain only part of an object, so an additional blob may be collected to correspond to that target, the examiner is interpreting this as two blobs/contours may be collected for a single object (first contour and third contour) and depth information may be collected accordingly for each blob/contour (at least a first and second depth));
determine, based on the depth data associated with the first image at a second depth, a fourth contour associated with the at least one other object (Zhang, [0048] the system detects multiple blobs (contours) corresponding to multiple objects tracked, a blob is a group of pixels corresponding to an object, or a mask of an object in the image which can be used to distinguish targets in an image from one another, [0061] and [0062 further note these blobs are defined by boundary lines on the object, The examine notes that in the applicants specification, page Auto-exclusion zones section, the applicant defines the contours as a curve associated with the edge representation of a person or object, as well as these contours having pixel coordinates, therefore the blobs of Zhang would be analogous to this as described in [0048], [0052] depth data may be extracted from each blob (contour), where one or more blobs (contour) corresponds to one or more real world objects, a single blob (contour) may be determined to correspond to two different objects, further a blob (contour) may contain only part of an object, so an additional blob may be collected to correspond to that target, the examiner is interpreting this as two blobs/contours may be collected for a single object (first contour and third contour) and depth information may be collected accordingly for each blob/contour (at least a first and second depth), further given that a plurality of blobs are detected for a plurality of objects, multiple objects may be within the same blob, in which the system would separate those into separate contours later, included within the same blob, but corresponding to separate objects, indicating at least a second and a fourth contour/blob and corresponding depth information),
wherein the third contour and the fourth contour are determined within the single contour (Zhang, [0052] depth data may be extracted from each blob (contour), where one or more blobs (contour) corresponds to one or more real world objects, a single blob (contour) may be determined to correspond to two different objects, further a blob (contour) may contain only part of an object, so an additional blob may be collected to correspond to that target, the examiner is interpreting this as two blobs/contours may be collected for a single object (first contour and third contour) and depth information may be collected accordingly for each blob/contour (at least a first and second depth), further given that a plurality of blobs are detected for a plurality of objects, multiple objects may be within the same blob, in which the system would separate those into separate contours later, included within the same blob, but corresponding to separate objects, indicating at least a second and a fourth contour/blob and corresponding depth information, since multiple objects can exist within a single blob, and then be separated into individual contours, the examiner is interpreting this as the contours may exist within one contour/curve);
determine candidate identifiers for the tracked first object, wherein the candidate identifiers comprise a subset of identifiers of all tracked objects (Siddiquie (col 2 lines 35-53) teaches comparing the embedded vector of features of the object being tracked that may or may not need to be re-identified with other embedded vectors of other tracked agents/objects. Therefore, the subset that is the embedded vectors of the identifier of the “first object” are compared to other identifier embedded vectors of other tracklet tracked objects/agents);
determine, based at least in part on the received first image and the third contour, an updated identifier for the tracked first object, wherein the updated identifier is one of the candidate identifiers (Siddiquie, column 4 line 54 through column 5 line 45, the foreground and background segmentations (plurality of contours, including at least a 1st-third contour) are used in re-identification of the agents in the video, where the embedding vectors are the updated identifiers);
determine, from the first image, that a portion of the first object has crossed a second predefined zone, wherein the second predefined zone defines a virtual curtain in front of the rack (Fisher, column 23 lines 1-17, the cameras use a time series of images (at least a first image) to determine if an object is picked up from a shelf and removed, where the distance of the person’s hand from the shelf is a known distance/area, this is being interpreted as being analogous to a second predefined zone in front of a shelf, column 23 line 49- 58 the shelf (zone 1) is mapped and the areas in front of the shelves and the rest of the store are mapped as well (at least a second zone in front of the shelf), whether or not items have been removed from the shelves is tracked, therefore it would be determined if an item exits the shelf (zone 1) and enters the region in front of the shelf (zone 2), Figure 14 and pages 4 and 5 of applicant’s specification define the virtual curtain as a zone in front of the rack which the objects pass through to determine if they have been removed from a rack or shelf. );
PNG
media_image5.png
170
320
media_image5.png
Greyscale
(Fisher, column 23, emphasis added)
and in response to determining that the portion of the first object has crossed the second predefined zone, assign the updated identifier to the tracked first object (Fisher, column 23 lines 1-17, the cameras use a time series of images (at least a first image) to determine if an object is picked up from a shelf and removed, where the distance of the person’s hand from the shelf is a known distance/area, this is being interpreted as being analogous to a second predefined zone in front of a shelf, column 23 line 49- 58 the shelf (zone 1) is mapped and the areas in front of the shelves and the rest of the store are mapped as well (at least a second zone in front of the shelf), whether or not items have been removed from the shelves is tracked, therefore it would be determined if an item exits the shelf (zone 1) and enters the region in front of the shelf (zone 2), once an object is determined as being removed from the shelf (entering zone 2) its index number is updated (updated identifier is assigned)).
The combination of Siddiquie, Zhou, Zhang and Fisher would be obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. The motivation for the combination lies in that the addition of the ability to merge “blobs” or contours to aide in the identification of objects or persons on video would have improved the system of Siddiquie in that it would reduce mis-identification of persons on the video (Zhou Column 1, lines 58-67, Column 11, lines 53-58, Column 1 lines 67- Col 2 Lines 1-3, Column 17, lines 29-44). Further, the combination of Siddiquie, Zhou and Zhang would be obvious to one of ordinary skill in the art prior to the effective filing date of the presently claimed invention. The combination of Siddiquie and Zhou fails to teach the determination of a third and fourth contour with depth information belonging to a single contour. In the same field of endeavor of object tracking and re-identification, Zhang teaches this deficiency. The motivation to add this feature of Zhang lies in that the inclusion of more comprehensive depth information to the detection capacities of Siddiquie and Zhou allows for features of the person being tracked, such as height or size to be more accurately detected, as well as more accurate 3D position detection (Zhang [0042]- [0045]. The motivation for the addition of the determination that the object enters a second zone in front of the shelf as taught by Fisher is that it allows for accurate tracking of inventory items which have been removed from the shelf by a person for ease of virtual payment. (Fisher, column 23)
Regarding claim 16 the combination of Siddiquie, Zhou, Zhang and Fisher teach;
The tracking subsystem of Claim 15, wherein the processor of the tracking subsystem is further configured to (Siddiquie Column 4 lines 55-60, A processor unit is configured to operate capabilities of the system):
determine, based on the first image, a first feature value associated with observable characteristics of the tracked first object (Siddiquie (col 4 line 52 through col 5 line 20) discloses assigning to each agent (each agent corresponding to a worker/user/person/object) a feature value that is an embedding vector (col 5 lines 1-15, embedding vector EV110 corresponding to 1st agent, EV120 to 2nd agent, EV130 to 3rd agent). Siddiquie (col 5 lines 35-45) has further disclosed that these embedding vectors describe observable object attributes including but not limited to color, size, shape, texture. The resultant embedding vectors assigned to each agent (180) or tracklet to produce feature sets (188) (col 6 lines 20-35));
compare the first feature value to a set of predetermined feature values associated with the candidate identifiers determined for the tracked first object (Siddiquie (col 6 lines 30-43) discloses the set of new and old embedded vector type feature values are compared to determine similarity with an associated agent, hence embedded vector of image data for an agent/object is compared to one or more candidate identifier “embedded vectors” for the first and so on tracked agent/person/object of Siddiquie);
based on results of the comparison, determine the updated identifier for the tracked first object, wherein the updated identifier is the predetermined feature value from the set of predetermined feature values with a value that is within a threshold range of the first feature value (Siddiquie, (col 2 lines 45-67) discloses selecting a “first object” candidate that is one of the multiple objects (agents) in the image data who has a “candidate identifier” most similar to the previous identifier type embedded vector of the Agent_1/Agent_A. New embedded vector data of the identifier for the Agent_1/”first object” is determined based on the similarity determination of the candidate identifiers (col 2 lines 45-67). Siddiquie (col 6 lines 25-40) has disclosed replacing or further including the identifier/”embedding vector”/features calculated for the agent/object).
Regarding claim 17 the combination of Siddiquie, Zhou and Zhang teach;
The tracking subsystem of Claim 16, wherein: the first feature value comprises a first data vector associated with the observable characteristics of the tracked first object (Siddiquie (col 4 line 52 through col 5 line 20) discloses assigning to each agent 180-1 - 180-4 (each agent corresponding to a person/object) a feature value that is an embedding vector (col 5 lines 1-15, embedding vector EV110 corresponding to 1st agent, EV120 to 2nd agent, EV130 to 3rd agent, and so on). Siddique (col 5 lines 35-45) has further disclosed that these embedding vectors describe observable object attributes including but not limited to color, size, shape, texture. The resultant embedding vectors assigned to each agent 180 or tracklet to produce feature sets 188 (col 6 lines 20-35). The embedding vector for an agent comprising at least some “first” value else no value would be assigned to said corresponding agent);
each of the predetermined feature values from the set of predetermined feature values comprises a corresponding predetermined data vector (Siddiquie, Col 4, line 52 – col 5 line 20 teaches at least one embedding vector associated with each respective agent/object);
and the processor of the tracking subsystem is further configured to (Siddiquie, Column 3, Lines 55-66, the system has a processor unit (134)): compare the first feature value to each of the predetermined feature values from the set of predetermined feature values associated with the candidate identifiers by calculating a similarity value between the first data vector and each of the predetermined data vectors (Siddiquie (col 2 lines 45-67) discloses selecting a “first object” that is one of the plurality objects in the image data who has a “candidate identifier” most similar to the previous identifier type embedded vector of the Agent_1/Agent_A. New embedded vector data of the identifier for the Agent_1/”first object” is determined based on the similarity determination of the candidate identifiers (col 2 lines 45-67). Siddiquie (col 6 lines 25-40) has disclosed replacing or further including the identifier/”embedding vector”/features calculated for the agent/object);
and determine the updated identifier as a particular candidate identifier that corresponds to the similarity value that is nearest one (Siddiquie (col 6 lines 20-43) discloses updating weighting and inclusion of new embedding vectors “identifiers” based on a similarity comparison determination of closeness of match of said feature type embedding vectors).
Regarding claim 18 the combination of Siddiquie, Zhou, Zhang and Fisher teach;
The tracking subsystem of Claim 17, wherein the processor of the tracking subsystem is further configured to determine that each of the similarity values is less than a threshold similarly value and, in response (Siddiquie, Column 19 lines 64-column 20 lines 1-9, The system generates a similarity score for the agent (object) identified, and confidence score, features are compared, the confidence score is then determined to exceed a threshold to determine the agent, Examiner is interpreting this as the system has the ability to determine if a similarity threshold exceeds or falls below a certain level):
determine a second feature value for each of the one or more other objects, wherein each second feature value comprises a second data vector (Siddiquie (col 4 line 52 through col 5 line 20) discloses assigning to each of agent (each agent corresponding to a worker/user/person/object) a feature value that is an embedding vector (col 5 lines 1-15, embedding vector EV110 corresponding to 1st agent, EV120 to 2nd agent, EV130 to 3rd agent, and so on). Siddique (col 5 lines 35-45) has further disclosed that these embedding vectors describe observable object attributes including but not limited to color, size, shape, texture, The resultant embedding vectors assigned to each agent 180 or tracklet to produce feature sets 188 (col 6 lines 20-35). Multiple embedding vectors, which would include second, third, and so on feature values are associated with the one or more agents (col 5 lines 50-67, col 6 lines 1-10, col 6 lines 20-43, col 6 lines 50-65, col 7 lines 1-31));
determine a second similarity value between each of the second data vectors and each of the predetermined feature values (Siddiquie (col 4 line 52 through col 5 line 20) discloses assigning to each of agent (each agent corresponding to a worker/user/person/object) a feature value that is an embedding vector (col 5 lines 1-15, embedding vector EV110 corresponding to 1st agent, EV120 to 2nd agent, EV130 to 3rd agent, and so on). Siddique (col 5 lines 35-45) further discloses that these embedding vectors describe observable object attributes including but not limited to color, size, shape, texture. The resultant embedding vectors assigned to each agent 180 or tracklet to produce feature sets 188 (col 6 lines 20-35). Multiple embedding vectors, which would include a second and third feature values are associated with the one or more agents (col 5 lines 50-67, col 6 lines 1-10, col 6 lines 20-43, col 6 lines 50-65, col 7 lines 1-31));
and determine second updated identifiers for the tracked first object and each of the one or more other objects, based on the first and second similarity values (Siddiquie discloses (col 4 line 52 through col 5 line 20, col 5 lines 35-45, col 5 lines 50-67, col 6 lines 1-10, col 6 lines 20-43, col 6 lines 50-65, col 7 lines 1-31 as cited above) performing a plurality of similarity determinations (Including at least a first and second) for first and second embedded vectors for first second and so on agents/objects in order to determine and update a set of feature value/”embedded vectors”/identifiers for the plurality of agents).
Regarding claim 19 the combination of Siddiquie, Zhou, Zhang and Fisher teach; The tracking subsystem of Claim 15, wherein the tracked first object is a first person (Siddiquie (col 8 lines 20-35) discloses using a plurality of digital imaging devices to capture overlapping fields of view of data in a facility (col 2 lines 7-37). The image devices suppling image data to track users/objects/workers/agents in a facility using the disclosed computing device of Fig 4 (“tracklet” of the one or more agents – col 2 lines 7-37), the tracklet showing the tracked object is a person).
Regarding claim 20 the combination of Siddiquie, Zhou, Zhang and Fisher teaches; The tracking subsystem of Claim 16, wherein the processor (Siddiquie, Column 3, Lines 55-66, the system has a processor unit (134)) of the tracking subsystem is further configured to, prior to determining that re- identification of the tracked first object is needed, periodically determine updated predetermined feature values associated with the candidate identifiers (Siddiquie (col 6 lines 20-30) has disclosed periodically updating the embedded vectors (candidate identifiers) that are the feature sets identifying the objects/agent being tracking the image processing system).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. For a listing of analogous art as cited by the Examiner, please see the attached PTO-892 Notice of References Cited.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JORDAN M ELLIOTT whose telephone number is (703)756-5463. The examiner can normally be reached M-F 8AM-5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached on (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.M.E./Examiner, Art Unit 2666 /Molly Wilburn/Primary Examiner, Art Unit 2666
Prosecution Insights