DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 4-8, 9-10, 12-16, 17-18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over “Automatic threat recognition of prohibited items at aviation checkpoints with x-ray imaging: a deep learning approach” (hereinafter “Liang”, published 2018) in view of US PG Pub. 2023/0047697 A1 (hereinafter “Cella”).
Regarding claim 1, Liang teaches a computer-implemented method (Liang, Section 3.4; “The object detection models are trained on labeled data with a NVIDIA Titan X GPU”), the method comprising:
accessing a first video output stream from an internal scanning device as the internal scanning device scans one or more objects or people (Liang, Section 2.1; “Data used for the training and testing of the ATR methods were collected…using a Smiths Detection host x-ray system. The platform is a cabinet x-ray security system that contains four separate pairs of 160 KeV x-ray sources and detector arrays, arranged opposite each other around a tunnel. Passenger bags and other belongings are carried through the tunnel by a conveyor belt at a rate of 240 mm/sec…Image slices of a given view are assembled to produce two grayscale images corresponding to the two energies.”);
identifying the targeted subject matter in the first video output stream based on interactions by a human operator with portions of the first video output stream; generating a training dataset based on the identified targeted subject matter and corresponding portions of images in the first video output stream (Liang, Section 2.2; “Labeling of the training set was performed manually using a Smiths Detection proprietary utility to draw an outline around each threat object. These outlines were then used to create a binary mask of the threat object's location…the min and max coordinates of these masks were used to generate bounding boxes to train the model”); and
training a machine-learned model using the generated training dataset, the machine-learned model trained to identify the targeted subject matter in video streams from internal scanning devices (Liang, Section 3.1; “Each of the aforementioned object detection models, termed meta-architectures, along with several different CNNs have been implemented in TensorFlow20 as part of Google's Object Detection API,13 and we leverage these implementations as the basis for our experiments…We train each model entirely on data from a single class”).
Liang does not expressly teach the limitations as further claimed, but, in an analogous field of endeavor, Cella does as follows.
Cella teaches accessing a second video output stream from an image-capturing device configured to record a human operator as the human operator reviews the first video output stream displayed on a display and interacts with portions of the first video output stream to identify targeted subject matter, which in turn generates the second video output stream (Cella, ¶0097; “an artificial intelligence/machine learning system may be trained on a training set of data that consists of tracking and recording sets of interactions of humans as the humans interact with a set of interfaces, such as graphical user interfaces (e.g., via interactions with mouse, trackpad, keyboard, touch screen, joystick, remote control devices)…the RPA system may record mouse clicks on a frame of video that appears within a process by which a human review the video, such as where the human highlights points of interest within the video, tags objects in the video, captures parameters (such as sizes, dimensions, or the like), or otherwise operates on the video within a graphical user interface.”).
Cella is considered analogous art because it pertains to training a model to detect objects of interest marked by human review. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method taught by Liang to include recording human operator interactions with a user interface highlighting points of interest or tagging objects in system-recorded video, as taught by Cella, in order to more accurately train the object detection model to more effectively and accurately replicate the human action of tagging or labeling an object of interest (Cella, ¶0097).
Regarding claim 2, claim 1 is incorporated, and Liang in the combination further teaches wherein the internal scanning device is one of an MRI scanner, an X-ray scanner, a CAT scanner, or a backscatter scanner (Liang, Section 2.1; “Data used for the training and testing of the ATR methods were collected…using a Smiths Detection host x-ray system.”).
Regarding claim 4, claim 1 is incorporated, and Liang in the combination further teaches: applying the machine-learned model to a target video output stream from a target internal scanning device to identify the targeted subject matter; modifying the target video output stream to include indications of the identified targeted subject matter; and displaying the modified target video output stream (Liang, Sections 3.1-3.4, Fig. 4; “The end goal, however, is to deliver a system that can be deployed to airport security checkpoints. Therefore, initial laboratory prototypes pipe scanned images to a GPU and computer bolted to the exterior of the x-ray scanner to compute threat locations, which are then sent back and projected on top of the color images on the main display.”).
Regarding claim 5, claim 4 is incorporated, and Cella in the combination further teaches receiving an indication from a target human operator that the identified target subject matter is a false positive; generating a new training dataset based on the indication; and retraining the machine-learned model based on the new training dataset (Cella, ¶0097; “Through a large training set of observation of human interactions and system states, events, and outcomes, the RPA system may learn to interact with the system in a fashion that mimics that of the human. Learning may be reinforced by training and supervision, such as by having a human correct the RPA system as it attempts in a set of trials to undertake the action that the human would have undertaken (e.g., tagging the right object, labeling an item correctly, selecting the correct button to trigger a next step in a process, or the like), such that over a set of trials the RPA system becomes increasingly effective at replicating the action the human would have taken. Learning may include deep learning, such as by reinforcing learning based on outcomes”).
As established above with respect to claim 1, Cella is considered analogous art because it pertains to training a model to detect objects of interest marked by human review. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method taught by Liang to include human feedback to correct the output of the detection model and reinforce the learning of the model based on the correction, as taught by Cella, in order to more accurately train the object detection model to more effectively and accurately replicate the human action of tagging or labeling an object of interest (Cella, ¶0097).
Regarding claim 6, claim 4 is incorporated, and Cella in the combination further teaches receiving an indication from a target human operator that confirms the identified target subject matter as correct; generating a new training dataset based on the indication; and retraining the machine-learned model based on the new training dataset (Cella, ¶0097; “Learning may be reinforced by training and supervision, such as by having a human correct the RPA system as it attempts in a set of trials to undertake the action that the human would have undertaken (e.g., tagging the right object, labeling an item correctly, selecting the correct button to trigger a next step in a process, or the like), such that over a set of trials the RPA system becomes increasingly effective at replicating the action the human would have taken.”).
As established above with respect to claim 1, Cella is considered analogous art because it pertains to training a model to detect objects of interest marked by human review. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method taught by Liang to include human feedback to validate the output of the detection model and reinforce the learning of the model based on the correction, as taught by Cella, in order to more accurately train the object detection model to more effectively and accurately replicate the human action of tagging or labeling an object of interest (Cella, ¶0097).
Regarding claim 7, claim 4 is incorporated, and Cella in the combination further teaches receiving an indication from a target human operator that the identified target subject matter within the modified target video output stream was missed; generating a new training dataset based on the indication; and retraining the machine-learned model based on the new training dataset (Cella, ¶0097; “Learning may be reinforced by training and supervision, such as by having a human correct the RPA system as it attempts in a set of trials to undertake the action that the human would have undertaken (e.g., tagging the right object, labeling an item correctly, selecting the correct button to trigger a next step in a process, or the like), such that over a set of trials the RPA system becomes increasingly effective at replicating the action the human would have taken.”).
As established above with respect to claim 1, Cella is considered analogous art because it pertains to training a model to detect objects of interest marked by human review. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method taught by Liang to include human feedback to correct the output of the detection model and reinforce the learning of the model based on the correction, as taught by Cella, in order to more accurately train the object detection model to more effectively and accurately replicate the human action of tagging or labeling an object of interest (Cella, ¶0097).
Regarding claim 8, claim 4 is incorporated, and receiving an indication from a target human operator that modifies the identified target subject matter; generating a new training dataset based on the indication; and retraining the machine-learned model based on the new training dataset (Cella, ¶0097; “Learning may be reinforced by training and supervision, such as by having a human correct the RPA system as it attempts in a set of trials to undertake the action that the human would have undertaken (e.g., tagging the right object, labeling an item correctly, selecting the correct button to trigger a next step in a process, or the like), such that over a set of trials the RPA system becomes increasingly effective at replicating the action the human would have taken.”).
As established above with respect to claim 1, Cella is considered analogous art because it pertains to training a model to detect objects of interest marked by human review. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method taught by Liang to include human feedback to correct the output of the detection model and reinforce the learning of the model based on the correction, as taught by Cella, in order to more accurately train the object detection model to more effectively and accurately replicate the human action of tagging or labeling an object of interest (Cella, ¶0097).
Claim 9 recites a non-transitory computer-readable medium storing instructions having features corresponding to the steps of method claim 1. Therefore, the recited elements of Claim 9 are mapped to the proposed combination in the same manner as the corresponding elements in Claim 1. Additionally, the rationale and motivation to combine the Liang and Cella references presented in the rejection of Claim 1 apply to this claim, and Liang in the combination further teaches a non-transitory computer-readable storage medium storing executable instructions executed by a hardware processor (Liang, Section 3.4).
Claim 10 recites a non-transitory computer-readable medium storing instructions having features corresponding to the steps of method claim 2. Therefore, the recited elements of Claim 10 are mapped to the proposed combination in the same manner as the corresponding elements in Claim 2. Additionally, the rationale and motivation to combine the Liang and Cella references presented in the rejection of Claim 2 apply to this claim.
Claim 12 recites a non-transitory computer-readable medium storing instructions having features corresponding to the steps of method claim 4. Therefore, the recited elements of Claim 12 are mapped to the proposed combination in the same manner as the corresponding elements in Claim 4. Additionally, the rationale and motivation to combine the Liang and Cella references presented in the rejection of Claim 4 apply to this claim.
Claim 13 recites a non-transitory computer-readable medium storing instructions having features corresponding to the steps of method claim 5. Therefore, the recited elements of Claim 13 are mapped to the proposed combination in the same manner as the corresponding elements in Claim 5. Additionally, the rationale and motivation to combine the Liang and Cella references presented in the rejection of Claim 5 apply to this claim.
Claim 14 recites a non-transitory computer-readable medium storing instructions having features corresponding to the steps of method claim 6. Therefore, the recited elements of Claim 14 are mapped to the proposed combination in the same manner as the corresponding elements in Claim 6. Additionally, the rationale and motivation to combine the Liang and Cella references presented in the rejection of Claim 6 apply to this claim.
Claim 15 recites a non-transitory computer-readable medium storing instructions having features corresponding to the steps of method claim 7. Therefore, the recited elements of Claim 15 are mapped to the proposed combination in the same manner as the corresponding elements in Claim 7. Additionally, the rationale and motivation to combine the Liang and Cella references presented in the rejection of Claim 7 apply to this claim.
Claim 16 recites a non-transitory computer-readable medium storing instructions having features corresponding to the steps of method claim 8. Therefore, the recited elements of Claim 16 are mapped to the proposed combination in the same manner as the corresponding elements in Claim 8. Additionally, the rationale and motivation to combine the Liang and Cella references presented in the rejection of Claim 8 apply to this claim.
Claim 17 recites a system having elements corresponding to the steps of method claim 1. Therefore, the recited elements of Claim 17 are mapped to the proposed combination in the same manner as the corresponding elements in Claim 1. Additionally, the rationale and motivation to combine the Liang and Cella references presented in the rejection of Claim 1 apply to this claim, and Liang in the combination further teaches a system comprising a computer processor and a non-transitory memory storing executable instructions (Liang, Section 3.4).
Claim 18 recites a system having elements corresponding to the steps of method claim 2. Therefore, the recited elements of Claim 18 are mapped to the proposed combination in the same manner as the corresponding elements in Claim 2. Additionally, the rationale and motivation to combine the Liang and Cella references presented in the rejection of Claim 2 apply to this claim.
Claim 20 recites a system having elements corresponding to the steps of method claim 4. Therefore, the recited elements of Claim 20 are mapped to the proposed combination in the same manner as the corresponding elements in Claim 4. Additionally, the rationale and motivation to combine the Liang and Cella references presented in the rejection of Claim 4 apply to this claim.
Claims 3, 11 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Liang in view of Cella, as applied to claims 1, 9 and 17 above, and further in view of “A vehicle threat detection system using correlation analysis and synthesized X-ray images” (hereinafter “Zheng”, published 2013).
Regarding claim 3, claim 1 is incorporated, and Liang in the combination further teaches wherein the internal scanning device is an X-ray scanner configured to scan items at a security checkpoint to identify at least one of the following targeted subject matters: drugs, weapons, or explosives (Liang, Introduction, Section 2; “In this context, threats refer to items prohibited by the TSA, which can vary widely, including (but not limited to) firearms, sharps, blunt weapons, precursors, and explosives.”).
Neither Liang nor Cella in the combination expressly teaches wherein the X-ray scanner is configured to scan a vehicle at a security checkpoint, but, in an analogous field of endeavor, Zheng does as follows.
Zheng teaches wherein the internal scanning device is an X-ray scanner configured to scan vehicles at a security checkpoint to identify targeted subject matters (Zheng, Introduction, last paragraph, Section 3.1; “To address the needs of automated threat detection at borders, we propose a vehicle threat detection system utilizing synthesized X-ray images and temporal image analysis. The proposed system provides a comprehensive solution of vehicle threat detection” wherein “In a real application, the X-ray images are acquired via real-time vehicle scanning, which will be processed and used as probe images.”).
Zheng is considered analogous art because it pertains to threat detection at security checkpoints. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the method taught by the combination of Liang in view of Cella to be implemented at a vehicle security checkpoint configured with an X-ray scanner to scan vehicles for abnormal goods, as taught by Zheng, in order to automatically and accurately recognize dangerous smuggling goods (Zheng, Section 4).
Claim 11 recites a non-transitory computer-readable medium storing instructions having features corresponding to the steps of method claim 3. Therefore, the recited elements of Claim 11 are mapped to the proposed combination in the same manner as the corresponding elements in Claim 3. Additionally, the rationale and motivation to combine the Liang, Cella and Zheng references presented in the rejection of Claim 3 apply to this claim.
Claim 19 recites a system having elements corresponding to the steps of method claim 3. Therefore, the recited elements of Claim 19 are mapped to the proposed combination in the same manner as the corresponding elements in Claim 3. Additionally, the rationale and motivation to combine the Liang, Cella and Zheng references presented in the rejection of Claim 3 apply to this claim.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. The additional references provided pertain to security checkpoint processes typically performed by human operators, and/or automating said processes.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAMAH A BEG whose telephone number is (571)270-7912. The examiner can normally be reached M-F 9 AM - 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, HENOK SHIFERAW can be reached at 571-272-4637. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SAMAH A BEG/Primary Examiner, Art Unit 2676
Prosecution Insights