DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-15 are pending.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claim(s) 1-2, 4-5, 8-12, 14-15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Harvill et al, US Patent Pub US 20200050965 A1 (hereinafter Harvill) in view of Vagman et al, US Patent Pub US 20150248755 A1 (hereinafter Vagman) in view of Kim et al, US Patent Pub US 20210374477 A1 (hereinafter Kim)
Claim 1
Harvill teaches a method for commissioning an artificial intelligence (AI) based inspection system, comprising: (i) receiving, by a commissioning computer, sensor data captured by a sensor (Harvill, para 20, 25 – Receiving image/sensor data from a camera/sensor by a computer.), (ii) collecting, by the commissioning computer, a plurality of data samples, wherein collecting each data sample comprises providing an operator interface for a field operator to access sensor data pertaining to individual items and thereto assign classification labels, the collected data samples corresponding to a fixed setting of the sensor (Harvill, para 50-51 – Collecting data samples with hand labeling/”field operator assigning classification labels” that correspond to camera/sensor settings.), (iii) training, by the commissioning computer, an AI algorithm using the collected data samples to configure the AI algorithm from input sensor data (Harvill, para 74-75 – Training a machine learning model/”AI algorithm” using the collected/”input sensor” data samples.), (iv) testing, by the commissioning computer, the trained AI algorithm by feeding the trained AI algorithm with real-time sensor data captured by the sensor (Harvill, para 74-77 – Testing the machine learning model/”AI algorithm” using real data captured by the camera/sensor.), wherein steps (ii) through (iv) are performed iteratively using different fixed settings of the sensor, to determine a final setting of the sensor for which a defined success criterion is achieved during the testing (Harvill, para 39, 49, 74, 91 – Iteratively training and testing using changed hyperparameter/”different fixed settings” for the camera/sensor and determining a success criteria.), and (v) deploying the iteratively trained and tested AI algorithm to a field device, the field device being coupled to the sensor configured with the final setting to provide sensor data as input to the deployed AI algorithm. (Harvill, para 94 – Making/deploying the trained model for use by the camera/sensors to provide data used in training.)
But Harvill fails to specify the sensor positioned within a physical environment to capture sensor data pertaining to individual items in a sequence of similar items.
However Vagman teaches the sensor positioned within a physical environment to capture sensor data pertaining to individual items in a sequence of similar items. (Vagman, para 60-64 – Sensors positioned in a physical manufacturing environment to capture sensor data from a sequence of same objects/”similar items”.)
Harvill and Vagman are analogous art because they are from the same field of endeavor. They relate to inspection systems.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the above inspection system, as taught by Harvill, and incorporating the above limitations, as taught by Vagman.
One of ordinary skill in the art would have been motivated to do this modification in order to determine possible errors of the measured object by incorporating the above limitations, as suggested by Vagman (para 4).
But the combination of Harvill and Vagman and fails to specify configure the AI algorithm to predict classification labels from input sensor data.
However Kim teaches configure the AI algorithm to predict classification labels from input sensor data. (Kim, para 42-43 – Using a neural network/”AI algorithm” in training an image classification model to predict a label of the input image/”sensor data”.)
Harvill, Vagman, and Kim are analogous art because they are from the same field of endeavor. They relate to inspection systems.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the above inspection system, as taught by Harvill and Vagman, and incorporating the above limitations, as taught by Kim.
One of ordinary skill in the art would have been motivated to do this modification in order to allow the image classification model to have high image classification performance with only a small number of labeled images by incorporating the above limitations, as suggested by Kim (para 5).
This rejection also applies to claim 15.
Claim 2
The combination of Harvill, Vagman, and Kim teaches all the limitations of the base claims as outlined above.
Vagman further teaches the physical environment comprises a production line or an assembly line. (Vagman, para 38, Fig 2 refs(2) – The environment of the system is a production line assembling objects.)
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the above inspection system, as taught by Harvill and Vagman, and incorporating the above limitations, as taught by Kim.
One of ordinary skill in the art would have been motivated to do this modification in order to determine possible errors of the measured object by incorporating the above limitations, as suggested by Vagman (para 4).
Claim 4
The combination of Harvill, Vagman, and Kim teaches all the limitations of the base claims as outlined above.
The combination of Harvill, Vagman, and Kim further teaches each fixed setting of the sensor is defined by a combination of internal sensor settings and/or arrangements of the sensor in relation to the physical environment. (Harvill, para 25 – Internal settings, positions and alignment/”arrangements of the sensor in relation to the physical environment” for the camera/sensor.)
Claim 5
The combination of Harvill, Vagman, and Kim teaches all the limitations of the base claims as outlined above.
The combination of Harvill, Vagman, and Kim further teaches the sensor comprises a vision camera capable of transmitting captured sensor data as a video stream to the commissioning computer and to the field device. (Harvill, para 25, 104-105 – A video camera is used as the sensor.)
Claim 8
The combination of Harvill, Vagman, and Kim teaches all the limitations of the base claims as outlined above.
The combination of Harvill, Vagman, and Kim further teaches the sensor comprises an acoustic sensor capable of transmitting captured sensor data as an audio stream to the commissioning computer and to the field device. (Harvill, para 104 – Input devices/sensor including a microphone/”acoustic sensor”.)
Claim 9
The combination of Harvill, Vagman, and Kim teaches all the limitations of the base claims as outlined above.
The combination of Harvill, Vagman, and Kim further teaches the sensor comprises a vibration sensor. (Harvill, para 104 – Input devices/sensor including an accelerometer/”vibration sensor”.)
Claim 10
The combination of Harvill, Vagman, and Kim teaches all the limitations of the base claims as outlined above.
The combination of Harvill, Vagman, and Kim further teaches storing the collected data samples in a local storage medium of the commissioning computer. (Harvill, para 28, 30, 37 – Collected data stored in a local storage medium of the computer.)
Claim 11
The combination of Harvill, Vagman, and Kim teaches all the limitations of the base claims as outlined above.
The combination of Harvill, Vagman, and Kim further teaches the training and the testing of the AI algorithm is carried out via a graphics processing unit (GPU) of the commissioning computer. (Harvill, para 99 - Processing information and instructions using a graphics processing unit (GPU).)
Claim 12
The combination of Harvill, Vagman, and Kim teaches all the limitations of the base claims as outlined above.
The combination of Harvill, Vagman, and Kim further teaches converting the trained AI algorithm, by the commissioning computer, into a format compatible with the field device, such that the testing by the commissioning computer is carried out on the converted format of the AI algorithm. (Harvill, para 93, 111 - The trained model is converted to target execution platform as in mobile/field devices.)
Claim 14
The combination of Harvill, Vagman, and Kim teaches all the limitations of the base claims as outlined above.
The combination of Harvill, Vagman, and Kim further teaches a non-transitory computer-readable storage medium including instructions that, when processed by a computing device, configure the computing device to perform the method. (Harvill, para 101 – A non-transitory computer-readable medium that may be used to store instructions and data which when executed by the processor cause performing computer-implemented methods to execute the techniques.)
Claim(s) 3, 6-7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Harvill et al, US Patent Pub US 20200050965 A1 (hereinafter Harvill) in view of Vagman et al, US Patent Pub US 20150248755 A1 (hereinafter Vagman) in view of Kim et al, US Patent Pub US 20210374477 A1 (hereinafter Kim) as applied to claims 1-2, 4-5, 8-12, 14-15 above, and in further view of Guo et al, US Patent Pub US 20210248427 A1 (hereinafter Guo).
Claim 3
The combination of Harvill, Vagman, and Kim teaches all the limitations of the base claims as outlined above.
But the combination of Harvill, Vagman, and Kim and fails to specify the AI algorithm is pre-trained prior to training by the commissioning computer, wherein the iterative training and testing via the commissioning computer is carried out to tune the pre-trained AI algorithm to said sensor and said physical environment.
However Guo teaches the AI algorithm is pre-trained prior to training by the commissioning computer, wherein the iterative training and testing via the commissioning computer is carried out to tune the pre-trained AI algorithm to said sensor and said physical environment. (, para 37 - Pre-training modules to generate initial or default neural networks using the image/sensor data in its environment.)
Harvill, Vagman, Kim, and Guo are analogous art because they are from the same field of endeavor. They relate to inspection systems.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the above inspection system, as taught by Harvill, Vagman, and Kim, and incorporating the above limitations, as taught by Guo.
One of ordinary skill in the art would have been motivated to do this modification in order to reduce the time and expense of training a neural network by incorporating the above limitations, as suggested by Guo (para 2).
Claim 6
The combination of Harvill, Vagman, and Kim teaches all the limitations of the base claims as outlined above.
But the combination of Harvill, Vagman, and Kim fails to specify the operator interface comprises a camera viewer configured to display live sensor data to the field operator, the method further comprising adjusting a current setting of the sensor by the field operator based on the displayed live sensor data prior to collection of the data samples.
However Guo teaches the operator interface comprises a camera viewer configured to display live sensor data to the field operator, the method further comprising adjusting a current setting of the sensor by the field operator based on the displayed live sensor data prior to collection of the data samples. (Guo, para 38 – A user/”field operator” views the video sequence input allowing the user to adjust annotations/”current setting” of the camera sensor based on the video in a setup/”prior to collection” mode.)
Harvill, Vagman, Kim, and Guo are analogous art because they are from the same field of endeavor. They relate to inspection systems.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the above inspection system, as taught by Harvill, Vagman, and Kim, and incorporating the above limitations, as taught by Guo.
One of ordinary skill in the art would have been motivated to do this modification in order to reduce the time and expense of training a neural network by incorporating the above limitations, as suggested by Guo (para 2).
Claim 7
The combination of Harvill, Vagman, and Kim teaches all the limitations of the base claims as outlined above.
But the combination of Harvill, Vagman, and Kim fails to specify the operator interface comprises a frame grabber for capturing a still image from the video stream, the operator interface configured to enable the field operator to draw a bounding box around a region of interest in the captured image and label the captured image.
However Guo teaches the operator interface comprises a frame grabber for capturing a still image from the video stream, the operator interface configured to enable the field operator to draw a bounding box around a region of interest in the captured image and label the captured image. (Guo, para 38 – A user/”field operator” views the video sequence input allowing the user to adjust annotations/”current setting” of the camera sensor based on the video frames in a setup/”prior to collection” mode.)
Harvill, Vagman, Kim, and Guo are analogous art because they are from the same field of endeavor. They relate to inspection systems.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the above inspection system, as taught by Harvill, Vagman, and Kim, and incorporating the above limitations, as taught by Guo.
One of ordinary skill in the art would have been motivated to do this modification in order to reduce the time and expense of training a neural network by incorporating the above limitations, as suggested by Guo (para 2).
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Harvill et al, US Patent Pub US 20200050965 A1 (hereinafter Harvill) in view of Vagman et al, US Patent Pub US 20150248755 A1 (hereinafter Vagman) in view of Kim et al, US Patent Pub US 20210374477 A1 (hereinafter Kim) as applied to claims 1-2, 4-5, 8-12, 14-15 above, and in further view of Samsung, “Samsung Electronics to Strengthen its Neural Processing Capabilities for Future AI Applications”, June 2019, Samsung Newsroom, pp 1 (hereinafter Samsung).
Claim 13
The combination of Harvill, Vagman, and Kim teaches all the limitations of the base claims as outlined above.
But the combination of Harvill, Vagman, and Kim and fails to specify the field device comprises a neural processing unit (NPU).
However Samsung teaches the field device comprises a neural processing unit (NPU). (Samsung, p1 – A mobile/field device equipped with an NPU.)
Harvill, Vagman, Kim, and Samsung are analogous art because they are from the same field of endeavor. They relate to machine learning systems.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the invention to modify the above machine learning system, as taught by Harvill, Vagman, and Kim, and incorporating the above limitations, as taught by Samsung.
One of ordinary skill in the art would have been motivated to do this modification in order to provide advanced on device AI features by incorporating the above limitations, as suggested by Samsung (p1).
Citation of Pertinent Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Choi et al, US Patent Pub US 20220083855 A1 relates to claims regarding generating synthetic data based on a pre-trained machine learning model, audio and video data processing, and labeling data.
Metzler et al, US Patent Pub US 20220157138 A1 relates to claims regarding a plurality of surveillance sensors configured to provide data about one or more facility elements, neural networks, audio and video sensors.
Visser et al, US Patent Pub US 20170270406 A1 relates to claims regarding training a neural network based on the spatial information labels and sensor data from audio and video inputs, and neural processing units.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID E OGG whose telephone number is (469) 295-9163. The examiner can normally be reached on Mon - Thurs 7:30 am - 5:00 pm CT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mohammad Ali can be reached on 571-272-4105. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID EARL OGG/
Primary Examiner, Art Unit 2119