DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-12 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claim(s) 1 and 7-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kudo et al (US20190012621) in view of Milan et al (Multiple Object Tracking, 2013) and further in view of Raghavan et al (US10475185).
Regarding claim 1, 10 and 11, Kudo teaches an image analysis device comprising:
an input interface configured to acquire image data indicating a captured image of a site in which a plurality of operators performs a plurality of tasks;
(Kudo, Fig. 1, “The first measurement device G1 and the second measurement device G2 are devices that acquire sensing data used for analysis by the information processing device 100. The first measurement device G1 is a network camera for capturing movement of a worker in a wide range (that is, photographing a distant view), and is installed at a work site. The second measurement device G2 is a network camera for capturing detailed movement of the worker (that is, photographing a near view), and is installed at the work site. The first measurement device G1 and the second measurement device G2 transmit a momentary photographed result to the information processing device 100 via a network N based on a predetermined communication protocol”, [0031]; providing an input acquisition interface for multi-operator task environments)
a controller configured to generate, based on the image data, task history information indicating the tasks performed in the site by respective operators among the plurality of operators; and
(Kudo, Fig. 1, “a work data management system of the present invention includes a detailed step classification unit that classifies work performed by a worker at a work site into a plurality of time-series detailed steps based on work data including image data of the work site and a data visualization unit that causes a display unit to display at least the image data and the plurality of time-series detailed steps”, [0008]; “The working motion statistical analysis unit 23 also has a function of generating a flow line history of a predetermined part based on the momentary position of the predetermined part (head and hand) of the worker”, [0070]; a controller/output generating time-series history mapped to specific operators)
a storage configured to store the task history information,
(Kudo, Figs. 1-2, “The measurement data storing unit 12 writes the image data described above in a memory area and outputs predetermined image data to the data visualization unit 24 according to a read instruction from the data visualization unit 24”, [0024]; “The detailed step storing unit 17 successively writes detailed step data in the memory area and outputs the detailed step data to the data visualization unit 24 according to a read instruction from the data visualization unit 24”, [0061]; Storage of both image data and classified/detail task history; “detailed step data” => history)
wherein the controller is configured to:
successively recognize the tasks and positions of the plurality of operators, based on the image data at each time in the site; and
(Kudo, “The event determination unit 16a successively determines momentary “events” of work at the work site based on time-series feature quantity data input from the feature quantity extraction unit 13 and outputs the determination result to the step determination unit 16c”, [0047]; “The feature quantity extraction unit 13 extracts a feature quantity of image data input to itself via the measurement interface 11 and specifies a position of the head and hands of a worker”, [0039])
Kudo does not expressly disclose but Milan teaches:
detect crossing caused between a plurality of trajectories, each trajectory including successive positions for each of the plurality of operators, and
(Milan, Fig. 1; “modeling mutual exclusion between distinct targets be-comes important at two levels: (1) in data association, each target observation should support at most one trajec-tory and each trajectory should be assigned at most one observation per frame; (2) in trajectory estimation, two trajectories should remain spatially separated at all times to avoid collisions”, p3682:c1; “We specifically address mutual exclusion both at the data-association and at the trajectory level (cf . Fig. 1)” (p3683:c1) ... “To ensure that only one of two overlapping trajectories is suppressed, the penalty was added only to one trajectory in each competing pair” (p3684:c2) ... “We now turn to the pairwise label cost ... The energy should be high if there exist two labels that are unlikely to appear simultaneously ... the co-occurrence penalty is proportional to the spatio-temporal overlap between the two trajectories” (p3685:c1); penalizes trajectory crossing (overlap), and detects such events at the modeling level)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the teachings of Milan into the system or method of Kudo in order to analyze and avoid workflow bottlenecks in a work environment due to workers’ overlapping trajectories. The combination of Kudo and Milan also teaches other enhanced capabilities.
The combination of Kudo and Milan further teaches:
wherein the controller configured to:
generate the task history information by associating tasks recognized at each time with the respective operators, based on the plurality of trajectories, when the crossing is not detected; and
(Kudo, “the detailed step classification unit 16 (see FIG. 2) classifies the work performed by the worker at the work site into a plurality of time-series detailed steps based on work data including the image data of the work site”, [0058]; “In the example illustrated in FIG. 5, the date and time at which photographing of the work site was performed, the detailed step, the elapsed time from the start of the detailed step, the position of the head of the worker, and the position of the hands of the worker are included in the working motion data”, [0067]; association of step/task history and trajectory (positions, head/hands) when tracking is unambiguous)
The combination of Kudo and Milan does not expressly disclose but Raghavan teaches
associate the recognized tasks with the respective operators
(Raghavan, "When there are multiple users near an event location, the implementations described herein provide the ability to disambiguate between the multiple users and determine which user performed the event.", c2:45-50; "If it is determined that there are multiple user patterns within the defined distance of the event location during the event time window, the user disambiguation process 1600 is performed to determine which user pattern is to be associated with the event.", c23:1-10; "determine if there is an occlusion or an area near the event location that is blocked from the field of view of the cameras", c30:30-35; detecting a crossing or occlusion (multiple users/operators physically occupying or occluding the same space near an event) and associating the recognized task/event with the correct operator/user)
... by selecting a combination from candidate combinations of the recognized tasks and the recognized operators of which positions are recognized for the plurality of trajectories causing the crossing,
(Raghavan, "If there are multiple candidate user patterns for an event, the implementations described herein may disambiguate between candidate user patterns and determine which user pattern is to be associated with the event.", c21:65-c24:5; "If it is determined that there are multiple events within the defined distance during the event time window, the probability scores for each user pattern with respect to each event are obtained using the example process 1300", c31:45-50; "For example, if there are three users (user A, user B, user C) that had initial probabilities of 88%, 84%, 82%, respectively, for a first event, one or more of those probabilities may be adjusted based on the confidence scores generated for a second event.", c31:55-65; "To illustrate, if user C has the highest probability for a second event and a confidence score of 98% for the second event, it may be determined that user B is associated with the second event. Accordingly, the probability that user B is also involved in and should be associated with the first event may be reduced because it is unlikely that user B was involved in both events.", c31:60-c32:5; generating and evaluating candidate combinations of recognized tasks (multiple events) and recognized operators (multiple candidate user patterns). When trajectories cross, these combinations are probabilistically scored to select the correct user-to-event association and eliminate conflicting combinations)
... based on the recognized tasks and past task history information generated on performed tasks by the respective operators ...
(Raghavan, "In addition to considering the arm trajectory and user orientation of the user pattern, user historical information may also be considered to confirm that the user pattern is to be associated with the event, as in 1310.", c24:20-25; "For example, the user historical information, such as which arm is typically utilized by the user to pick items, whether the user has previously picked the item involved in the event, the items picked by the user during the current session at the materials handling facility, etc. may be considered as a factor in determining the probability that the user associated with the user pattern was involved in the event.", c24:25-35; relying on past task history generated by the operator (e.g., historical events, past items picked during the current session) to calculate the probability score used to associate the current recognized task with the operator)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to incorporate the history-based and probability-based disambiguation process of Raghavan into the modified image analysis device of Kudo and Milan in order to accurately resolve identity tracking ambiguities when multiple operators' trajectories cross or occlude one another near a task location, ensuring that workflow metrics and task histories remain correctly assigned to the respective operators rather than failing during crowded conditions. The combination of Kudo, Milan and Raghavan also teaches other enhanced capabilities.
Regarding claim 7, the combination of Kudo, Milan and Raghavan further teaches its/their respective base claim(s).
The combination further teaches the image analysis device according to claim 1, wherein the controller is configured to detect the crossing based on overlapping of positions of two or more operators among the plurality of operators in the image indicated by the acquired image data.
(Milan, “The resulting trajectories have to explain the observations in a physically plausible way, i.e... trajectories must not overlap, because two objects cannot occupy the same physical space at the same time. ... the second challenge is thus approached at the trajectory level. We introduce a novel pairwise co-occurrence label cost”, p3684:c1; “the co-occurrence penalty is proportional to the spatio-temporal overlap between the two trajectories ... which is computed by summing the mutual overlap over all frames in the common lifespan O of the trajectories”, p3685:c1; This means the detection of crossing is specifically based on overlapping positions of two or more operators (targets) in the image data; detecting trajectory crossings by measuring when the positions of two or more operators (targets) overlap in the image data, using a penalty that depends on the amount of spatio-temporal overlap between their tracks)
Regarding claim 8, the combination of Kudo, Milan and Raghavan further teaches its/their respective base claim(s).
The combination further teaches the image analysis device according to claim 1, wherein
the storage is further configured to store identification information which identifies the respective operators, and
the controller is configured to associate recognized positions with the respective operators when the positions of the operators in the site are recognized for first time.
(Milan, “each detection must be assigned a target identifier or discarded as a false alarm”, p3683:c1; “Here it is important to not only enforce a unique assignment for each detection, but also to constrain that two simultaneous observations must not be assigned to the same target”, p3684:c1; operating on detections and assigning each detection a target identifier; Milan teaches both storing identification (track identities) and associating positions with operators (target ID) when they are first detected in the image data, via its CRF detection labeling)
Regarding claim 9, the combination of Kudo, Milan and Raghavan further teaches its/their respective base claim(s).
The combination further teaches the image analysis device according to claim 1, wherein the controller is configured to generate information indicating a ratio of the plurality of tasks performed over a predetermined period, for each of the operators, based on the task history information in the predetermined period.
(Kudo, Figs. 1 and 6A; 64, “The working time statistical analysis unit 20 calculates a predetermined statistical value regarding the working time of the detailed step ... performed in the past predetermined period... The working time statistical data includes, for example, the minimum value, the first quartile, the median value, the third quartile, and the maximum value of the working time in the past week”, [0064]; “the data visualization unit 24 causes the result display unit 26 to display at least image data and a plurality of time-series detailed steps (Gantt chart P1 and the like). With this, the user (administrator) can grasp at a glance how much time the worker took for each of the detailed steps”, [0076]; “Based on the statistical values, the box-and-whisker plot P2 of the detailed step A1 is created and superimposed and displayed on the Gantt chart P1”, [0077]; the controller aggregates and statistically analyzes the time and occurrence of each detailed operator task over selected time periods, displaying the results for each operator as ratios (e.g., median, mean, distribution) using Gantt charts and statistical summaries)
Allowable Subject Matter
Claim 12 is allowed.
Claim(s) 2-6 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening Claim(s).
The following is a statement of reasons for the indication of allowable subject matter:
Claim(s) 2-6 recite(s) the limitation(s) directed to storing site task tendencies to accurately match recognized tasks to operators during crossing. No explicit teachings are found in the prior art cited in this office action and from the prior art search.
Response to Arguments
Applicant's arguments filed on 1/19/2026 with respect to one or more of the pending claims have been fully considered but are moot in view of the new ground(s) of rejection.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIANXUN YANG whose telephone number is (571)272-9874. The examiner can normally be reached on MON-FRI: 8AM-5PM Pacific Time.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center. for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272- 1000.
/JIANXUN YANG/
Primary Examiner, Art Unit 2662 4/4/2026
Prosecution Insights