DETAILED ACTION
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine,
manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-8, 17, 18, and 21-30 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. When reviewing independent claim 1, and based upon consideration of all of the relevant factors with respect to the claim as a whole, claims 1-8, 17, 18, and 21-30 are held to claim an abstract idea without reciting elements that amount to significantly more than the abstract idea and is/are therefore rejected as ineligible subject matter under 35 U.S.C. 101.
The Examiner will analyze Claim 1, and similar rationale applies to independent Claim 17 and Claim 18.
The rationale, under MPEP § 2106, for this finding is explained below. The claimed invention (1) must be directed to one of the four statutory categories, and (2) must not be wholly directed to subject matter encompassing a judicially recognized exception, as defined below. The following two step analysis is used to evaluate these criteria.
Step 1: Is the claim directed to one of the four patent-eligible subject matter categories: process, machine, manufacture, or composition of matter?
When examining the claim under 35 U.S.C. 101, the Examiner interprets that the claims is related to a process since the claim is directed to a method of recognizing a package sorting behavior.
Step 2a, Prong 1: Does the claim wholly embrace a judicially recognized exception, which includes laws of nature, physical phenomena, and abstract ideas, or is it a particular practical application of a judicial exception?
The Examiner interprets that the judicial exception applies since Claim 1 limitation of performing a target detection on an image frame in a target video, so as to acquire at least one human detection box and at least one package detection box [a person can recognize human and package in a video]; tracking a motion trajectory of the human detection box and a motion trajectory of the package detection box respectively [a person can also track the human and the package in a video]; recognizing, in a process of trajectory tracking for a package, a moment at which the package is thrown based on a current motion trajectory tracked for the package; and acquiring motion information of the package from the moment at which the package is thrown to a current moment, and recognizing a sorting behavior of the package based on the motion information are directed to an abstract. The claim is related to mental process by having a person reviewing a video to determine mishandling of a package.
If/when the claim recites a judicial exception (i.e., an abstract idea enumerated in MPEP § 2106.04(a), a law of nature, or a natural phenomenon), the claim requires further analysis in Prong Two.
Step 2a, Prong 2: Does the claim recite additional elements that integrate the judicial exception into a practical application? NO
Step 2b: If a judicial exception into a practical application is not recited in the claim, the Examiner must interpret if the claim recites additional elements that amount to significantly more than the judicial exception.
Furthermore, the generic computer components or machine learning algorithm of the processor/memory/method recited as performing generic computer or machine learning functions that are well-understood, routine and conventional activities amount to no more than implementing the abstract idea with a computerized system.
The Examiner finds that Claims 2-8, and 21-30 does not state significantly more since the claim only recites additional steps for analyzing video to determine mishandling of a package.
Thus, claims 1-8, 17, 18, and 21-30 recite the same abstract idea and therefore are not drawn to the eligible subject matter as they are directed to the abstract idea without significantly more.
Therefore, all claims are rejected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 2, 4, 6, 17, 18, 21, 23, 25, 28, and 30 are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Zhang et al. (Pub. No. US 2023/0063926).
Regarding claim 1, Zhang et al. (Pub. No. US 2023/0063926) teaches a method of recognizing a package sorting behavior (rough handling), comprising: performing a target detection (detects) on an image frame (video frame) in a target video (video stream), so as to acquire at least one human (person) detection box and at least one package detection box [Para. 137 “In certain aspects, the present disclosure relates to a method for recognizing or monitoring rough handling”; “As shown in FIG. 7, at procedure 702, the video capture module 140 instructs the imaging device 110 to take a video stream of an area of interest at real time, and sends the video stream to video decoder 142”; and Para. 141 “At procedure 708, upon receiving the t-th frame from the video decoder 142, the detection module 146 detects person bounding boxes and keypoints and package bounding boxes from the t-th frame, and sends the person bounding boxes and keypoints and package bounding boxes to the postprocessing module 148.”]; tracking a motion trajectory (person tracking trajectories) of the human detection box (person bounding boxes) and a motion trajectory (package tracking trajectories) of the package detection box (package bounding boxes) respectively [Para. 143 “At procedure 712, upon receiving the person bounding boxes and keypoints and motion package bounding boxes from the postprocessing module 148, the tracking module 150 tracks the persons and the motion packages to obtain or update person tracking trajectories and package tracking trajectories, and sends the updated person tracking trajectories and package tracking trajectories to the action recognition module 152”]; recognizing, in a process of trajectory tracking for a package (package tracking trajectories), a moment at which the package is thrown (drop-off) based on a current motion trajectory tracked for the package (motion package tacking trajectories) [Para. 144 “At procedure 714, upon receiving the updated person tracking trajectories and package tracking trajectories from the tracking module 150, the action recognition module 152 update actions using the current (updated) person tracking trajectories and motion package tracking trajectories. Each action includes a number of consecutive person tracking positions (or frames) and the corresponding package tracking positions (or frames), starts from the person picks up the package, and ends a predetermined number of frames after the person drops off the package.”; Para. 152 “The action starts from frame t−x, where the person p.sub.i picks up the package q.sub.j; the person p.sub.i then holds the package q.sub.j for a while until frame t−1; at frame t, the person p.sub.i drops off the package q.sub.j; and the action ends after certain number of frames after the last hold or after the drop-off”]; and acquiring motion information (distances and speeds) of the package from the moment at which the package is thrown (drop-off) to a current moment (current frame), and recognizing a sorting behavior (rough handling) of the package based on the motion information (speeds and distances) [Para. 152 “the action recognition module 152 determines the rough handling based on multiple distances and speeds, and the distances and speeds are calculated starting from (t+2)-th frame, the third frame from the drop-off”. Para. 153 “Here the accumulated distance is the distance between the package in the current frame and the person's wrist keypoint in the last hold frame, the average speed is speed of the package in the last three frames (current frame and its two immediate previous frames), and the max speed is the greatest one at a frame from the (t+3)-th frame to the current frame”; Para. 152 “If the criteria are met, the action recognition module 152 ends the action and further calculates a rough handling score for the action; if the criteria are not met, the actions continues, and the action recognition module 152 will make a determination for the next frame, the (t+6)-th frame”].
Regarding claims 2, 21, and 28, Zhang teaches wherein the current motion trajectory comprises a human motion trajectory (person tracking trajectory) corresponding to a human detection box (person bounding box) and a package motion trajectory (package tracking trajectory) corresponding to a package detection box, and recognizing, for the package, the moment at which the package is thrown based on the current motion trajectory tracked for the package comprises: comparing a target package motion trajectory (package tracking trajectory) corresponding to the package with each human motion trajectory (person tracking trajectory), so as to acquire a target distance (distance) between a target package detection box (package bounding box) corresponding to the package and each human detection box (person bounding box) at a detected moment [Para. 91 “The tracking module 150 is configured to, upon receiving the detected person bounding boxes and keypoints and the motion package bounding boxes in the current frame, track the persons and the packages in parallel, and send the updated person trajectories and package trajectories to the action recognition module 152”; Para. 145 “For the person p.sub.i and the package q.sub.j in the t-th frame, which belong to the person tracking trajectory f.sub.i.sup.t and the package tracking trajectory g.sub.j.sup.t, the action recognition module 152 first calculates the distance between the person p.sub.i and the package q.sub.j and records the distance at the procedure 502.” And Para. 151 “The procedures are repeated for all the k number of persons P.sup.t={p.sub.1.sup.t, . . . , p.sub.k.sup.t} and all the l′ number of motion packages Q′.sup.t={q.sub.1.sup.t, . . . , q.sub.l′.sup.t}.”] and determining a moment at which the target distance is less than a distance threshold for a first time, as the moment at which the package is thrown [Para. 147 “If the person p.sub.i and the package q.sub.j do not belong to one of the actions, and there are person or package interpolations, the action recognition module 152 will begin the process from the earliest of the frame having the interpolation, for example the (t−3)-th frame, sets aside the person p.sub.i and the package q.sub.j if their distance is greater than the threshold at the (t−3)-th frame, and starts a new action for the person p.sub.i and the package q.sub.j if their distance at the (t−3)-th frame equals to is less than the threshold, where the status of the new action at t-th frame is “pick up””].
Regarding claims 4, 23, and 30, Zhang teaches wherein the determining a moment at which the target distance is less than a distance threshold for a first time, as the moment at which the package is thrown comprises: determining a moment corresponding to an image frame in which the target distance is less than the distance threshold [Para. 147, fig. 2 and related description].
Regarding claims 6 and 25, Zhang teaches any one of wherein the recognizing a sorting behavior of the package based on the motion information comprises: generating a sorting force parameter (rough handling score) of the package based on the motion information, and determining a sorting force level (high rough handling score, medium rough handling score, light rough handling score) of the sorting behavior of the package according to the sorting force parameter [Para. “At procedure 718, after an action is determined to be a rough action, the action recognition module 152 may optionally calculate a rough handling score for the action, and sends the determined rough handling action and its rough handling score to the output module 154”, and “In certain embodiments, the action recognition module 152 defines a high rough handing score of 3 and a light rough handling score of 1”].
Claims 17 and 18 are also rejected for the same reason as claim 1. Furthermore, Zhang teaches an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor to perform the claim limitations [fig. 1 and corresponding claim limitations].
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 3, 8, 22, 27, and 29 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (Pub. No. US 2023/0063926) further in view of Vajapey (WO 2019/217281).
Regarding claims 3, 22, and 29, Zhang teaches wherein the comparing a target package motion trajectory (package tracking) corresponding to the package with each human motion trajectory (person tracking trajectory) comprises: acquiring, from the target package motion trajectory, first position information (position) of the target package detection box on each image frame [Para. 91]; acquiring, from the human motion trajectory, second position information (position) of the human detection box on the same image frame [Para. 91, and 102]; and acquiring the target distance (distance) according to the first position information (position) and the second position information (position) of a corresponding moment [Para. 145].
However, Zhang teaches doesn’t explicitly teach in a reverse chronological order.
Vajapey teaches in having a reverse chronological order [Para. 47].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Zhang’s package tracking trajectory traversal by applying Vajapey’s reverse frame analysis (reverse sequential order) to inspect stored package positions from the current frame back toward earlier frames. This modification improves Zhang by enabling backward localization of the worker/package interaction point from an already-detected moving package, thereby reducing unnecessary forward scanning.
Regarding claims 8 and 27, Zhang teaches wherein the tracking a motion trajectory of the human detection box (person’s bounding box) and a motion trajectory of the package detection box respectively comprises: tracking the human detection box [Para. 143 “If a match is found, the person's bounding box from the t-th frame is added to the matched person trajectory as the latest position of the person trajectory”], so as to generate a human motion trajectory corresponding to the human detection box; and tracking the package detection box, so as to generate a package motion trajectory corresponding to the package detection box [Para. 143 “If a match is found, the package's bounding box from the t-th frame is added to the matched package trajectory as the latest position of the package trajectory”].
However, Zhang doesn’t explicitly teach about the rest of claim limitations.
VAJAPEY teaches tracking the human detection box (bounding region) based on first identification information (object identifier or ID) of the human detection box and tracking the package detection box based on second identification information of the package detection box [Para. 46].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Zhang’s person trajectory tracking by assigning Vajapey’s object identifier or ID to each worker bounding region used to form a person trajectory. This modification improves Zhang by preserving worker identity across frames, thereby improving association of a package sorting event with the correct sorter.
Claims 5 and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (Pub. No. US 2023/0063926) in view of Vajapey (WO 2019/217281) further in view of Bazarevsky et al. (Pub. No. US 2021/0174519).
Regarding claims 5 and 24, Zhang teaches acquiring a distance between the first position information (package bounding box) and the human hand position information (wrist keypoint), as the target distance [Para. 114].
However, Zhang in view of Vajapey doesn’t explicitly teach the rest of claim limitations.
Bazarevsky teaches wherein the acquiring the target distance according to the first position information and the second position information of a corresponding moment comprises: extracting/cropping an image region (image frame region) marked by the second position information (oriented bounding box) from an image frame corresponding to the second position information [Para. 43]; performing a human key point detection (keypoints) on the image region to acquire human hand position information (wrist keypoint) [Para. 43].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Zhang’s person-keypoint detection, modified by Vajapey, by applying Bazarevsk’s key-point localization within a cropped image frame region to obtain hand landmark positions for distance computation. This medication improves Zhang by localizing hand keypoints in a smaller, hand focused region, thereby improving computational efficiency and reducing false hand position dictations.
Claims 7 and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. (Pub. No. US 2023/0063926) further in view of Marty et al. (Pub. No. US 2022/0122269).
Regarding claims 7 and 26, Zhang teaches to any one of wherein the motion information comprises a distance value (accumulated distance), a maximum speed (max speed), and an average speed of the package to the current moment (current frame), as well as a motion speed (speed) and an acceleration of the package [Para. 153 “Here the accumulated distance is the distance between the package in the current frame and the person's wrist keypoint in the last hold frame, the average speed is speed of the package in the last three frames (current frame and its two immediate previous frames), and the max speed is the greatest one at a frame from the (t+3)-th frame to the current frame”; and Para. 152, “The calculation includes the reference (or compared) package horizontal distance d.sub.h/d.sub.torso, the reference package vertical distance d.sub.v/d.sub.torso, the reference direct package distance d.sub.pq/d.sub.torso, the person-package pixel distance d.sub.pq, the reference person-package distance d.sub.pq/d.sub.torso, the maximum speed, the average speed, and the acceleration”].
However, Zhang doesn’t explicitly teach the rest of claim limitation.
Marty teaches an average speed of the package from the moment at which the package is thrown (instant the ball is released) to the current moment (time step), and each moment from the moment (time step) at which the package is thrown to the current moment [Para. 261 “From the instant the ball is released from the server's hand, the trajectory of the ball is primarily affected by gravity”; Para. 268 “A database query may then performed to extract the parameters of interest, such as, but not limited to, flight distance, maximum height, final speed, angle, etc”; and “The database may contain the relevant variables at each time step—acceleration, velocity position, flight angle, etc.”].
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Zhang’s person-keypoint detection by applying Marty key-point localization within a cropped image frame region to obtain hand landmark positions for distance computation. This medication improves Zhang by localizing hand keypoints in a smaller, hand focused region, thereby improving computational efficiency and reducing false hand position dictations.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SOLOMON G BEZUAYEHU whose telephone number is (571)270-7452. The examiner can normally be reached on Monday-Friday 10 AM-7 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached on 313-446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-0101 (IN USA OR CANADA) or 571-272-1000.
/SOLOMON G BEZUAYEHU/ Primary Examiner, Art Unit 2666
Prosecution Insights