DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed November 17, 2025 have been fully considered but they are not persuasive.
Independent claims 1, 10 and 19 have been amended to recite, in part, “based on the analysis, count a number of times merchandise or a portion of the operator is detected crossinq an outer boundary of the first sub-area durinq a checkout process at the checkout apparatus”. Applicant argues that this limitation is not taught by Brakob or Chang.
This limitation is similar to the limitation recited in original claims 2, 11 and 20, which were rejected in the nonfinal Office Action as being unpatentable over Brakob in view of Chang and further in view of Mullins. However, Applicant has not addressed the merits of the rejections of claims 2, 11 and 20.
Regarding the Brakob reference, Applicant argues: “[i]In any event, Brakob certainly fails to teach or suggest any ‘counting a number of times merchandise or a portion of the operator is detected crossing an outer boundary of the first sub-area during a checkout process at the checkout apparatus’".
The examiner disagrees. As indicated in the nonfinal Office Action, the system of Brakob identifies scanning motions, determines when the operator has faked a scanning motion, determines when the operator accidentally misses a scan, determines when the operator pauses a scanning motion, determines how quickly the operator performs the scanning motion, and verifies when a scanning motion results in the merchandise being registered by the scanning device.
Furthermore, the system of Brakob also timestamps motions determined to be scanning motions as well as merchandise scans that have been registered. Using these timestamps, the system determines whether the scanning motion occurs within a threshold amount of time of the merchandise being scanned/registered and uses that information to determine whether the detected scanning motion corresponded to the registered merchandise scan (para. [0069]). Therefore, the system records the timestamped scanning motions and the timestamped merchandise scans in order to compare the timestamp of the scanning motion with the timestamp of the merchandise scan. This record is essentially a count of detected scanning motions and a count of the registered merchandise scans. Just because Brakob does not explicitly describe counting the number of times merchandise or a portion of the operator is detected, that does not mean Brakob does not maintain a count of the number of times these events are detected.
Since the system of Brakob identifies scanning motions, determines when the operator has faked a scanning motion, determines when the operator accidentally misses a scan, determines when the operator pauses a scanning motion, determines how quickly the operator performs the scanning motion, verifies when a scanning motion results in the merchandise being registered by the scanning device, and records timestamps of these events, Brakob necessarily counts the number of times merchandise and/or a portion of the operator is detected crossing an outer boundary of a first sub-area during a checkout process at the checkout apparatus.
However, as indicated in the nonfinal Office Action, Brakob does not explicitly disclose adjusting the boundary position between the first sub-area and the periphery area in the images based on the counted number of times that the operator or the merchandise is detected as crossing the outer boundary.
In the nonfinal Office Action, the examiner relied on Chang as disclosing adjusting the boundary position between a first sub-area and a periphery area in the images captured by a camera based on behavior of people being captured in the images. Chang discloses a camera (Figs. 1 and 2, camera 200) that has a control unit (para. [0056], Fig. 2, image processor 210) having a region setter (para. [0056], Fig. 2, region setter 204) that sets a boundary position in a captured image (paras. [0043], [0045], the boundary position corresponding to a region of interest (ROI) in the captured image data set by the region setter 204 for analysis by the image processor 210). The image data has a periphery (Fig. 4, image data I has a periphery) and the ROI has a boundary position (Fig. 4, dashed box labeled “ROI”) that divides the image I into one part corresponding to the ROI and another part corresponding to the region of the image in between the ROI and the periphery of the image I (paras. [0063]-[0064], Fig. 4). The part of the image that is within the ROI is cropped (Fig. 4, cropped image Ic) and the cropped image is analyzed to perform object detection (para. [0064]). The boundary position corresponding to the ROI can be adjusted based on a detected event in the image data (detected movement of a person) (Figs. 9A-10, paras. [0082] and [0089]-[0096]).
Chang makes these adjustments to reduce the amount of image processing that is performed by processing only ROIs rather than processing the entire image (para. [0111]: “[a] camera according to embodiments of the present disclosure may reduce an object detection time and the amount of calculations by detecting an object by analyzing only a region of interest in a high resolution image rather than the entire high resolution image”). This is the same reason the present invention adjusts the boundary (para. [0004] of the present specification).
However, the combined teachings of Brakob and Chang do not teach adjusting the boundary position based on the counted number of times the merchandise or the operator crosses the boundary. In the nonfinal Office Action, the examiner relied on Mullins as disclosing determining if the frequency at which (i.e., the number of times) a person performs a “sweep action” that crosses a masked area of a shelf in the captured image exceeds a threshold level (Fig. 15A, paras. [0134]-[0138], “[a] theft event may also be detected if a person in the depicted zone performs a sweep action within any portion of the shelf 1514, shelf 1516, and/or aisle 1515 covered by one of the masks 1520, 1522, or 1524 once or a threshold number of times within a certain time period”; the masked parts in the images act as virtual trip wires or fences, para. [0136], and therefore represent boundary positions in the captured images). If this behavior is detected as having occurred a threshold number of times within a threshold time period, i.e., at a frequency that exceeds a threshold level, an alarm notification is output to an outside system (para. [0004]).
Based on the combined teachings of Brakob, Chang and Mullins, it would have been obvious to one of ordinary skill in the art to which the claimed invention pertains before the effective filing date of the present disclosure to modify the control unit of Brakob as modified based on the teachings of Chang further based on the teachings of Mullins to determine whether the number of times that a scanning motion or merchandise is detected as crossing the outer boundary exceeds a threshold level, and if so, to adjust the boundary position to thereby adjust the ROI being processed so that the volume of data being processed is reduced.
Applicant also argues that Brakob does not disclose “any updates to or movements of a region of interest boundary or the like in the surveillance images supplied to the model for analysis "during the checkout process" as opposed to offline training. That is, the model is fixed/stagnant ‘during the checkout process’ and any changes or updates to any part of the model occur via the offline training process.”
The examiner disagrees. Brakob discloses that “[t]he computing system 102 can use one or more image processing and/or image extraction techniques to select the portion of the image data that depicts the checkout lane 100N and, more particularly, the scanning motion(s) of the customer (para. [0070]). Brakob states that the computing system “can also use one or more machine learning models that are trained to extract only the customer's movement (e.g., the customer's arms, hands, and/or torso) from the image data and use that extracted portion of the image data for further processing described throughout this disclosure” (Emphasis Added). The use of the word “can” in the above quote means that the disclosure of Brakob is not limited to using machine learning models to perform the “image processing and/or image extraction techniques to select the portion of the image data that depicts the checkout lane 100N” (para. [0070]). Nevertheless, there is no reason why the computing system 102 of Brakob could not be configured to adjust the boundary position in real-time during the checkout process based on the teachings of Chang since Chang teaches adjusting the boundary position in real-time and that doing so is beneficial because it reduces the amount of image data that has to be processed.
Claim Interpretation
The claims in this application are given their broadest reasonable interpretation (BRI) using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. BRIs for some of the claim terms are provided below. If Applicant believes that the BRIs are incorrect, Applicant should cite portions of the present disclosure that justify different BRIs.
The BRI of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f), because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such a claim limitation is “a control unit” in claim 1. Because this claim limitation is being interpreted under 35 U.S.C. 112(f), it is being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
The BRI for this limitation, based on para. [0031] of the present specification, is that it means a processor such as a central processing unit and memory such as read-only memory (ROM) and/or random-access memory (RAM) storing instructions that when executed by the processor cause it to perform the functions of the control unit, and equivalents thereof.
If applicant does not intend to have this limitation interpreted under 35 U.S.C. 112(f), applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-7, 10-16, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over U.S. Publ. Appl. No. 2024/0296677 A1 of Brakob et al. (hereinafter referred to as “Brakob”) in view of U.S. Publ. Appl. No. 2019/0370591 A1 of Chang et al. (hereinafter referred to as “Chang”) and further in view of U.S. Publ. Appl. No. 2024/0071191 A1 of Mullins (hereinafter referred to as “Mullins”).
Regarding claim 1, Brakob discloses an information processing device (Fig. 7, computing system 102) for image analysis of images from a camera (Fig. 1, camera 102) positioned to image an operation (para. 0051]) at a checkout apparatus (Fig. 1, POS terminal 103), the information processing device comprising:
a control unit (Fig. 7, modules 704-707) configured to:
perform an analysis of a first sub-area of images from a camera to detect a movement of an operator or merchandise at a checkout apparatus (paras. [0067]-[0071], Fig. 2, steps 204-212, the control unit 704-707 performs the steps of imaging a field of view (FOV) that includes the checkout lanes to determine lane the scan event occurred (step 204), identifying the checkout lane where the scan event occurred (step 208), and analyzing a first sub-area of the image data corresponding to an extracted portion of the image data that includes the scanning motion (step 210) to detect scanning motion (step 212));
based on the analysis, count a number of times merchandise or a portion of the operator is detected crossinq an outer boundary of the first sub-area durinq a checkout process at the checkout apparatus (as indicated above in the response to Applicant’s arguments, since the system of Brakob identifies scanning motions, determines when the operator has faked a scanning motion, determines when the operator accidentally misses a scan, determines when the operator pauses a scanning motion, determines how quickly the operator performs the scanning motion, verifies when a scanning motion results in the merchandise being registered by the scanning device, and records timestamps of these events, Brakob necessarily counts the number of times merchandise and/or a portion of the operator is detected crossing an outer boundary of a first sub-area during a checkout process at the checkout apparatus);
output a notification of non-registration if the analysis of the first sub-area of the images indicates that an item of merchandise has arrived at a post-registration storage place of the checkout apparatus from a pre-registration storage place of the checkout apparatus without passing through a registration region (para. [0072], Fig. 2, steps 212 and 214, if it is determined that the scanning event was not registered at step 212, an output is generated notifying employees or security personnel that scanning of the merchandise was not registered; as shown in Fig. 1, the scanning device 116, which corresponds to the registration region of the POS terminal, is located in between the shelf on which the operator places the basket of merchandise 124 and the bagging area such that the basket shelf and the bagging area correspond to the pre-registration storage place and the post-registration storage place, respectively); and
during the checkout process, adjusting the boundary position between the first sub-area and the periphery area in the images based on the counted number of times (Brakob does not explicitly disclose this limitation).
The BRI for “boundary position” is that it means an imaginary line that divides the image captured by a camera positioned above the POS terminal into at least a center part that includes the scanning area and a part that is in between the scanning area and a periphery of the image that includes the side of the POV terminal that includes the shelf on which the operator places the basket before the operator scans the merchandise. The BRI for adjusting the boundary position based the count is that it means that the imaginary line is moved toward the scanning area or away from the scanning area toward the periphery of the image based on the count. The BRIs are based on Fig. 5 and paras. [0049]-[0064] of the present disclosure.
In Brakob, the boundary position is an imaginary line in the captured image that divides the image into a part that includes the scanning area 116 and a part that is in between scanning area 116 and a periphery of the image that includes the shelf on which the operator places the basket 124 before the operator scans the merchandise. Scanning motions by the operator cause the operator and the merchandise to cross this boundary position. This crossing of the boundary line is detected and a determination is made as to whether this is an affirmative scanning motion.
As indicated above, Brakob does not explicitly disclose adjusting the boundary position between the first sub-area and the periphery area in the images based on the counted number of times the operator or the merchandise is detected as crossing the boundary position.
Chang, in the same field of endeavor, discloses adjusting the boundary position between a first sub-area and a periphery area in the images captured by a camera based on behavior of people being captured in the images. In particular, Chang discloses a camera (Figs. 1 and 2, camera 200) that has a control unit (para. [0056], Fig. 2, image processor 210) having a region setter (para. [0056], Fig. 2, region setter 204) that sets a boundary position in a captured image (paras. [0043], [0045], the boundary position corresponding to a region of interest (ROI) in the captured image data set by the region setter 204 for analysis by the image processor 210). The image data has a periphery (Fig. 4, image data I has a periphery) and the ROI has a boundary position (Fig. 4, dashed box labeled “ROI”) that divides the image I into one part corresponding to the ROI and another part corresponding to the region of the image in between the ROI and the periphery of the image I (paras. [0063]-[0064], Fig. 4). The part of the image that is within the ROI is cropped (Fig. 4, cropped image Ic) and the cropped image is analyzed to perform object detection (para. [0064]). The boundary position corresponding to the ROI can be adjusted based on a detected event in the image data (detected movement of a person) (Figs. 9A-10, paras. [0082] and [0089]-[0096]).
It would have been obvious to one of ordinary skill in the art to which the claimed invention pertains before the effective filing date of the present disclosure to modify the control unit of Brakob to incorporate features and methodology of the image processor 210 of Chang such that the control unit of Brakob adjusts the boundary position in the image data based on the operator or the merchandise detected as crossing the boundary line. A person of ordinary skill in the art would have been motivated to make the modification to reduce the amount of image processing that is performed by processing only ROIs rather than processing the entire image, as taught by Chang (para. [0111]: “[a] camera according to embodiments of the present disclosure may reduce an object detection time and the amount of calculations by detecting an object by analyzing only a region of interest in a high resolution image rather than the entire high resolution image”). The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods (modifying the software executed by the computing system 102 of Brakob) to yield predictable results.
The combined teachings of Brakob and Chang do not teach adjusting the boundary position based on the counted number of times the merchandise or the operator crosses the boundary position. Mullins, in the same field of endeavor, discloses determining if a person performs a “sweep action” that crosses a masked area of a shelf in the captured image a number of times that exceeds a threshold level (Fig. 15A, paras. [0134]-[0138], “[a] theft event may also be detected if a person in the depicted zone performs a sweep action within any portion of the shelf 1514, shelf 1516, and/or aisle 1515 covered by one of the masks 1520, 1522, or 1524 once or a threshold number of times within a certain time period”; the masked parts in the images act as virtual trip wires or fences, para. [0136], and therefore represent boundary positions in the captured images). If this behavior is detected as having occurred a threshold number of times within a threshold time period, i.e., at a frequency that exceeds a threshold level, an alarm notification is output to an outside system (para. [0004]).
It would have been obvious to one of ordinary skill in the art to which the claimed invention pertains before the effective filing date of the present disclosure to modify the control unit of Brakob as modified based on the teachings of Chang, further based on the teachings of Mullins, to determine whether the number of times that a scanning motion or merchandise is detected as crossing the outer boundary exceeds a threshold level, and if so, adjusting the boundary position. A person of ordinary skill in the art would have been motivated to make the further modification to ensure that ROIs are properly set and that suspicious activity including frequent sweeping actions is detected while also reducing the amount of image processing that needs to be performed by processing only ROIs rather than the entire image. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods (modifying the software executed by the computing system 102 of Brakob) to yield predictable results.
Regarding claim 2, Brakob discloses that the control unit is configured to perform a behavior detection as the analysis of the first sub-area to determine whether the operator’s hand has crossed the boundary position (Brakob discloses determining whether behavior in the form of a scanning motion has been detected (step 212) in the first sub-area, where the first sub-area corresponds to the extracted portion of the image data that includes the scanning motion; movements that can be considered in determining whether or not a scanning motion has been detected include, for example, the speed at which merchandise is moved across the scanning device (para. [0081]), how close the operator’s hand is to the scanning device when the motion is performed (para. [0083]), whether the operator pauses for a predetermined period of time during the scanning motion (para. [0081]); based on these types of detected motion, the computing system 102 determines whether the customer performed an affirmative scanning motion, faked a scanning motion, forgot to scan the product, and/or performed an accidental missed scan….” (para. [0071])).
However, the combined teachings of Brakob and Chang do not explicitly disclose moving the boundary position to broaden the first sub-area if the number of times that an arm or a hand of the operator moves across the boundary position exceeds a threshold level.
As indicated above, Mullins discloses determining whether the number of times that a person performs a “sweep action” that crosses a masked area of a shelf in the captured image exceeds a threshold level (Fig. 15A, paras. [0134]-[0138], “[a] theft event may also be detected if a person in the depicted zone performs a sweep action within any portion of the shelf 1514, shelf 1516, and/or aisle 1515 covered by one of the masks 1520, 1522, or 1524 once or a threshold number of times within a certain time period”; the masked parts in the images act as virtual trip wires or fences, para. [0136], and therefore represent boundary positions in the captured images). If this behavior is detected as having occurred a threshold number of times within a threshold time period, an alarm notification is output to an outside system (para. [0004]).
It would have been obvious to one of ordinary skill in the art to which the claimed invention pertains before the effective filing date of the present disclosure to further modify the control unit of Brakob as modified based on the teachings of Chang further based on the teachings of Mullins to determine if the number of times a scanning motion crosses the boundary position exceeds a threshold level within a predetermined period of time, and if so, to move the boundary position to broaden the ROI since Chang teaches broadening or narrowing the ROI as needed to only analyze regions of interest in the images. A person of ordinary skill in the art would have been motivated to make the modification to ensure that ROIs are properly set and suspicious behavior is detected while also reducing the amount of image processing that is performed by processing only ROIs rather than the entire image. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software executed by the computing system 102 of Brakob).
Regarding claim 3, as indicated above in the rejection of claim 2, Brakob discloses that the control unit is configured to perform a behavior detection (detection of a scanning motion) as the analysis of the first sub-area to determine whether the operator’s hand has crossed the boundary position. Brakob also discloses determining when the scanning motion results in registration of the merchandise being detected (Figs. 1 and 2, steps 144 and 214, respectively, paras. [0061] and [0071]). However, the combined teachings of Brakob and Chang do not explicitly disclose moving the boundary position to broaden the first sub-area if the number of times that merchandise moves across the boundary position exceeds a threshold level during a predetermined period of time.
As indicated above, Mullins discloses determining if the number of times that a person performs a “sweep action” that causes a high-valued merchandise item to cross a masked area of a shelf in the captured image has occurred exceeds a threshold level within a predetermined period of time (Fig. 15A, paras. [0134]-[0138], “[a] theft event may also be detected if a person in the depicted zone performs a sweep action within any portion of the shelf 1514, shelf 1516, and/or aisle 1515 covered by one of the masks 1520, 1522, or 1524 once or a threshold number of times within a certain time period.”; the masks in the images act as virtual trip wires or fences, para. [0136], and therefore represent boundary positions in the captured images; para. [0137]: “[f]or example, a sweep action may occur if a person reaches into a shelf at a first location, grabs one or more items between the first location and a second location, and pulls those item(s) from the shelf at the second location.”). If this behavior is detected as having occurred more than a threshold number of times within a threshold time period, i.e., at a frequency that exceeds a threshold level, an alarm notification is output to an outside system (para. [0004]).
It would have been obvious to one of ordinary skill in the art to which the claimed invention pertains before the effective filing date of the present disclosure to modify the control unit of Brakob as modified based on the teachings of Chang further based on the teachings of Mullins to determine if the counted number of times that the scanning motion causes merchandise to move across the boundary position exceeds a threshold level within a predetermined prior of time, and if so, to move the boundary position to broaden the ROI since Chang teaches broadening or narrowing the ROI as needed to only analyze regions of interest in the images. A person of ordinary skill in the art would have been motivated to make the modification to ensure that regions of interest are properly set while also reducing the amount of image processing that is performed by processing only ROIs rather than the entire image. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software executed by the computing system 102 of Brakob).
Regarding claim 4, Brakob discloses that the control unit is configured to perform a behavior detection (scanning motion) as the analysis of the first sub-area to determine whether the operator’s hand or arm crosses the boundary position during a predetermined interval of time. Conversely, Brakob discloses that the control unit is configured to perform a behavior detection (scanning motion) as the analysis of the first sub-area to determine whether the operator’s hand or arm does not cross the boundary position during a predetermined interval of time. (para. [0081]: “the computing system 102 can also train the model(s) to identify other motions and anomalies that may be indicative of theft behavior. For example, the computing system 102 can train the model(s) to identify motion cadence within one or more threshold lengths of time and classify the cadence as theft behavior or normal behavior”).
However, the combined teachings of Brakob and Chang do not explicitly disclose moving the boundary position to narrow the first sub-area if the number of times that the arm or a hand of the operator moves across the boundary position does not exceed the threshold during a predetermined interval of time.
As indicated above, Mullins discloses determining if a person performs a “sweep action” that crosses a masked area of a shelf in the captured image within a predetermined interval of time, and conversely, determining the number of times a person performs a “sweep action” and whether or not that number exceeds a threshold within the predetermined period of time (Fig. 15A, paras. [0134]-[0138], “[a] theft event may also be detected if a person in the depicted zone performs a sweep action within any portion of the shelf 1514, shelf 1516, and/or aisle 1515 covered by one of the masks 1520, 1522, or 1524 once or a threshold number of times within a certain time period”). If this behavior is detected as not having occurred within a predetermined time period, an alarm notification is not output to the outside system.
It would have been obvious to one of ordinary skill in the art to which the claimed invention pertains before the effective filing date of the present disclosure to modify the control unit of Brakob as modified based on the teachings of Chang further based on the teachings of Mullins to determine if the scanning motion does not move across the boundary position more than a threshold number of time within a predetermined time interval, and based on that determination, to move the boundary position to narrow the ROI since Chang teaches broadening or narrowing the ROI as needed to only analyze regions of interest in the images. A person of ordinary skill in the art would have been motivated to make the modification to ensure that regions of interest are properly set while also reducing the amount of image processing that is performed by processing only ROIs rather than the entire image. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods to yield predictable results (modifying the software executed by the computing system 102 of Brakob).
Regarding claim 5, the rejection of claim 2 applies mutatis mutandis to claim 5.
Regarding claim 6, as indicated above in the rejection of claim 4, Brakob discloses that the control unit is configured to perform a behavior detection (scanning motion) as the analysis of the first sub-area to determine whether the operator’s hand does not cross the boundary position during a predetermined interval of time (para. [0081]: “the computing system 102 can also train the model(s) to identify other motions and anomalies that may be indicative of theft behavior. For example, the computing system 102 can train the model(s) to identify motion cadence within one or more threshold lengths of time and classify the cadence as theft behavior or normal behavior”). Since Brakob also discloses determining if the scanning motion resulted in merchandise being registered by the scanning device (steps 144 and 212), Brakob also teaches determining whether no merchandise is detected as crossing the outer boundary during a predetermined interval of time.
However, the combined teachings of Brakob and Chang do not explicitly teach moving the boundary position to narrow the first sub-area if no merchandise is detected as moving across the outer boundary during a predetermined interval of time.
Mullins discloses determining if a person does not perform a “sweep action” that resulted in merchandise crossing a masked area of a shelf in the captured image within a predetermined interval of time (Fig. 15A, paras. [0134]-[0138], “[a] theft event may also be detected if a person in the depicted zone performs a sweep action within any portion of the shelf 1514, shelf 1516, and/or aisle 1515 covered by one of the masks 1520, 1522, or 1524 once or a threshold number of times within a certain time period.”para. [0137]: “[f]or example, a sweep action may occur if a person reaches into a shelf at a first location, grabs one or more items between the first location and a second location, and pulls those item(s) from the shelf at the second location”). Because Mullins teaches determining whether this behavior occurs within a predetermined time interval, it conversely determines whether this behavior is detected as not having occurred within a predetermined time interval.
It would have been obvious to one of ordinary skill in the art to which the claimed invention pertains before the effective filing date of the present disclosure to modify the control unit of Brakob as modified based on the teachings of Chang further based on the teachings of Mullins to determine if the scanning motion does not move merchandise across the boundary position within a predetermined time interval, and based on that determination, move the boundary position to narrow the ROI since Chang teaches broadening or narrowing the ROI as needed to only analyze regions of interest in the images. A person of ordinary skill in the art would have been motivated to make the modification to ensure that regions of interest are properly set and that suspicious activity is detected while also reducing the amount of image processing that is performed by processing only ROIs rather than the entire image. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods (modifying the software executed by the computing system 102 of Brakob) to yield predictable results.
Regarding claim 7, the rejection of claim 3 applies mutatis mutandis to claim 7.
Regarding claim 10, the rejection of claim 1 applies mutatis mutandis to claim 10.
Regarding claim 11, the rejection of claim 2 applies mutatis mutandis to claim 11.
Regarding claim 12, the rejection of claim 3 applies mutatis mutandis to claim 12.
Regarding claim 13, the rejection of claim 4 applies mutatis mutandis to claim 13.
Regarding claim 14, the rejection of claim 5 applies mutatis mutandis to claim 14.
Regarding claim 15, the rejection of claim 6 applies mutatis mutandis to claim 15.
Regarding claim 16, the rejection of claim 7 applies mutatis mutandis to claim 16.
Regarding claim 19, to the extent that claim 19 recites the same limitations that are recited in claim 1, the rejection of claim 1 applies mutatis mutandis to claim 19. The only limitation that is recited in claim 19 that is not also recited in claim 1 is a non-transitory, computer-readable storage medium storing program instructions which when executed by the control unit cause the control unit to perform the steps that are recited in claims 1 and 19. Brakob discloses a non-transitory, computer-readable storage medium storing program instructions for execution by the control unit to cause the control unit to perform the operations of Brakob that are described above in the rejection of claim 1 (paras. [0159]-[0160], Fig. 9).
Regarding claim 20, the rejection of claim 2 applies mutatis mutandis to claim 20.
Claims 8, 9, 17 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Brakob in view of Chang and Mullins as applied to claims 1-7, 10-16, 19 and 20 and further in view of U.S. Publ. Appl. No. 2011/0231419 A1 of Papke et al. (hereinafter referred to as “Papke”).
Regarding claim 8, the combined teachings of Brakob, Chang and Mullins do not explicitly disclose analyzing a second sub-area of the images, the second sub-area being adjacent to the first-sub area at the boundary position, to determine whether a person other than the operator is waiting to use the checkout apparatus.
Papke, in the same field of endeavor, discloses analyzing captured video at a retail location and in various zones within the retail location, including a checkout line (paras. [0009] and [0151]). The analysis of images can determine the flow of people in the different zones of the retail location and can determine the number of people in a zone such as a checkout line (para. [0064]).
It would have been obvious to one of ordinary skill in the art to which the claimed invention pertains before the effective filing date of the present disclosure to further modify the control unit of Brakob as modified based on the teachings of Chang and Mullins further based on the teachings of Papke to monitor a second sub-area such as the checkout line adjacent to the POS terminal to determine whether a person other than the operator is waiting to use the checkout apparatus as taught by Papke. A person of ordinary skill in the art would have been motivated to make the modification to better determine which images are associated with which customer’s behavior and to prevent detection of a theft event from being associated with the wrong person. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods (modifying the software executed by the computing system 102 of Brakob) to yield predictable results.
Regarding claim 9, the combined teachings of Brakob, Chang and Mullins do not teach this limitation. Papke teaches determining the number of customers that are in a particular zone, which can be a checkout area (para. [0082]: “[v]ideo analysis system 100 may be capable of determining or monitoring the occupancy of retail zone 310 throughout the day. Such may reflect a customer count representing a total number of customers present in an area or zone”).
It would have been obvious to one of ordinary skill in the art to which the claimed invention pertains before the effective filing date of the present disclosure to modify the control unit of Brakob as modified based on the teachings of Chang and Mullins further based on the teachings of Papke to monitor a second sub-area such as the checkout line adjacent to the POS terminal to determine the number of people other than the operator waiting to use the checkout apparatus as taught by Papke et al. A person of ordinary skill in the art would have been motivated to make the modification to better determine which images are associated with which customer’s behavior and to prevent detection of a theft event from being associated with the wrong person. The modification could have been made by one of ordinary skill in the art before the effective filing date of the present disclosure with a reasonable expectation of success because making the modification merely involves combining prior art elements according to known methods (modifying the software executed by the computing system 102 of Brakob) to yield predictable results.
Regarding claim 17, the rejection of claim 8 applies mutatis mutandis to claim 17.
Regarding claim 18, the rejection of claim 9 applies mutatis mutandis to claim 18.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Dan Santos whose telephone number is (571)272-2867. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matt Bella, can be reached on 571-272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Daniel J. Santos/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667