DETAILED ACTION
This action is written in response to the application filed 3/9/23. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
In determining whether the claims are subject matter eligible, the Examiner applies the 2019 USPTO Patent Eligibility Guidelines as well as MPEP § 2106.
Step 1: Is the claim to a process, machine, manufacture, or composition of matter? Yes—claim 1 recites a method, which is a process.
Step 2A, prong one: Does the claim recite an abstract idea, law of nature or natural phenomenon? Yes—the claim recites one or more limitations which—under their broadest reasonable interpretation—covers performance of the limitation in the mind (see table below).
Claim limitation
Examiner analysis
1. A method of identifying tasks associated with an industrial asset, the method comprising:
creating a first set of images, wherein the first set of images comprises location data associated with the plurality of images;
This is a mental process akin to a human evaluation/judgment/observation.
determining a plurality of target industrial assets based on the first set of images;
This is a mental process akin to a human evaluation/judgment/observation: visually identifying known equipment (assets).
creating a second set of images, wherein the second set of images comprises the process data and the first set of images;
This is a mental process akin to a human evaluation/judgment/observation.
identifying a target industrial asset within the plurality of target industrial assets based on the second set of images, wherein the target industrial asset is identified by determining that a degree of match between an image of the target industrial asset and at least one image from the plurality of images satisfies a predetermined threshold; and
This is a mental process akin to a human evaluation/judgment/observation: visually identifying known equipment (assets).
Because the claim recites limitations which can practically be implemented as mental processes, the claim recites a mental process.
Step 2A, prong two: Does the claim recite additional elements that integrate the judicial exception into a practical application? No—the claim does not recite even generic computer hardware, and is not directed to solving any particular real-world technological problem.
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No—the additional limitations are addressed below:
obtaining a plurality of images associated with the industrial asset;
This is insignificant pre-solution activity: gathering data to be processed in subsequent steps.
retrieving process data associated with the plurality of target industrial assets;
This is insignificant pre-solution activity: gathering data to be processed in subsequent steps.
based on identifying the target industrial asset, automatically retrieving a task associated with the target industrial asset.
This is insignificant post-solution activity: retrieving information about some unspecified task.
For the reasons above, claim 1 is rejected as being directed to non-patentable subject matter under §101. This rejection applies equally to independent claims 6 and 18, which recite a similar method and portable device, respectively, as well as to all pending dependent claims. The additional limitations of the dependent claims are addressed briefly below. Taken alone, the additional elements of the dependent claims above do not amount to significantly more than the above-identified judicial exception (the abstract idea). Looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide conventional computer implementation.
Claim limitation
Examiner analysis
2. The method of claim 1, wherein creating the first set of images comprises:
converting the location data to a first multi-dimensional dataset.
This is a mental process akin to a human evaluation/judgment.
3. The method of claim 1, wherein creating the second set of images comprises:
converting the process data to a second multi-dimensional dataset.
This is a mental process akin to a human evaluation/judgment.
4. The method of claim 1, further comprising creating a third set of images, wherein the third set of images comprises the second set of images and the process data associated with the plurality of target industrial assets.
This is merely additional information about one or more previously identified mental processes.
5. The method of claim 4, further comprising determining one or more measurement readings associated with the target industrial asset based on the third set of images.
This is a mental process akin to a human evaluation/judgment/observation.
10. The method of claim 6, wherein of the captured environmental input comprises at least one of a global positioning system (GPS) coordinate, a sensor reading, alarm data, equipment master data, and process data.
This is merely additional detail pertaining to the type of data received as input.
11. The method of claim 10, further comprising:
retrieving, from a database, one or more historical environmental inputs associated with the candidate industrial asset;
validating that the candidate industrial asset corresponds to a targeted industrial asset;
in response to validating that the candidate industrial asset corresponds to the targeted industrial asset, retrieving a task associated with the targeted industrial asset; and
automatically filling in at least one measurement reading for the task associated with the targeted industrial asset.
This is insignificant pre-solution activity: gathering data to be processed.
This is a mental process akin to a human evaluation/judgment/observation.
This is a mental process akin to a human evaluation/judgment/observation.
This is a mental process akin to a human evaluation/judgment/observation.
12. The method of claim 6, further comprising: retrieving historian data comprising historical readings associated with the candidate industrial asset; and merging the historian data with the captured image.
This is insignificant pre-solution activity:
gathering data to be processed.
13. The method of claim 12, further comprising:
predicting a reading for the candidate industrial asset based on the historian data; and
verifying the predicted reading for the candidate industrial asset matches an actual reading for the candidate industrial asset.
This is a mental process akin to a human evaluation/judgment/observation.
This is a mental process akin to a human evaluation/judgment/observation.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
The following are the references relied upon in the rejections below:
DeYoung (US 2020/0166904 A1)
Mikolajczyk (Mikolajczyk, Agnieszka, and Michał Grochowski. "Data augmentation for improving deep learning in image classification problem." In 2018 international interdisciplinary PhD workshop (IIPhDW), pp. 117-122. IEEE, 2018.)
Minisankar (US 2023/0057340 A1)
Schmidt (US 2023/0343066 A1)
Xiong (US 2022/0058591 A1, cited by Applicant in IDS dated 6/30/23.)
Claims 1-4, 6-7, 10, 12-14, 16 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over DeYoung and Schmidt.
Regarding claims 1, DeYoung discloses a method of identifying tasks associated with an industrial asset, the method comprising:
obtaining a plurality of images associated with the industrial asset;
[0027] “Computing system 101 identifies (203) the asset to be associated with the beacon 110, 112. In one example, a camera of computing system is accessed and operated to scan a bar code 115-118 associated with the respective asset 105-108.”
creating a first set of images, wherein the first set of images comprises location data associated with the plurality of images;
[0020] “The techniques disclosed herein attempt to facilitate identification of industrial equipment by including micro-location devices with individual pieces of equipment and processing the identification signal transmitted by the micro-location devices to correlate the identifiers to their related assets.”
determining a plurality of target industrial assets based on the first set of images;
[0027] “Computing system 101 identifies (203) the asset to be associated with the beacon 110, 112. In one example, a camera of computing system is accessed and operated to scan a bar code 115-118 associated with the respective asset 105-108. In an embodiment, the bar codes 115-118 may be designed to contain the identifying information for the associated asset 105-108. For example, the bar codes 115-118 may contain asset identification (ID) information, asset operation information, asset parameter information, and the like. In another embodiment, the bar codes 115-118 may provide an identifier information to computing system 101, which may then access an internal database or an external database (e.g., via communication link 121 and communication network 120 to an external database on application server 130) to acquire the information for the industrial asset 105-108 such as the information listed above.”
retrieving process data associated with the plurality of target industrial assets;
[0029] “ For example, an action rule may be set up such that when the computing system 101 is within a first distance threshold (e.g., 20 feet) of the beacon, the computing system 101 is operated in a monitoring mode. Accordingly, this action rule may cause a monitoring module to be automatically shown on the computing device 101 when within the first distance threshold that shows the key data values for all the devices in the cabinet containing the beacon. Another action rule may be set up such when the computing system 101 is within a second, closer distance threshold (e.g., 5 feet) of the beacon, maintenance is likely being performed. In this case, the computing device 101 may be caused to automatically show the work orders for the devices associated with the beacon 110, 112, the links to their manuals, or other maintenance information, for example. in this manner, the information displayed to the user may be reduced or expanded based on the proximity rules/actions associated with the asset. More than one action may be configured for or within a given time or distance threshold.”
[0020] “In an industrial automation environment, investigation is primarily dependent on available plant-wide information, which is typically accessible over a network of machines and other equipment. For example, technicians and other users commonly review tag data, trends, alarms, documentation, incident reports, chat transcripts, screen captures, and other content that is generated during the course of operating an industrial enterprise. However, this information is often not readily accessible when interacting with a machine, control system, or some other asset of the industrial operation. Thus, maintaining inventory, identifying equipment, and tracking assets in an industrial manufacturing environment can be difficult with existing solutions. Even once a piece of equipment has been identified, it may be challenging to correlate the equipment to the related software, logical and physical topology, status information, security access, documentation, and other content associated with the equipment. The techniques disclosed herein attempt to facilitate identification of industrial equipment by including micro-location devices with individual pieces of equipment and processing the identification signal transmitted by the micro-location devices to correlate the identifiers to their related assets.” (Emphasis added.)
creating a second set of images, wherein the second set of images comprises the process data and the first set of images; …
[0043] “ Examples of the information associated with asset 105-108 that could be retrieved by computing system 101 include operational data, machine data, images, screen graphics data, screen captures, video data, sound recordings, production processes, tag data, control information and logic, alarms, notifications, drive configurations, dashboards, human-machine interface (HMI) display screens, key performance indicators (KPIs), charts, trends, and other graphical content, simulation data, version numbers, catalogs, spare parts inventories, maintenance/repair schedules, links to documentation, electrical drawings, manuals, material safety data sheets, standard operating procedures, emergency procedures, cleanout procedures, reset procedures, safety procedures, incident reports including problems and solutions, chat transcripts, and any other information associated with the asset of the industrial environment.” (Emphasis added.)
based on identifying the target industrial asset, automatically retrieving a task associated with the target industrial asset.
[0029] “Another action rule may be set up such when the computing system 101 is within a second, closer distance threshold (e.g., 5 feet) of the beacon, maintenance is likely being performed. In this case, the computing device 101 may be caused to automatically show the work orders for the devices associated with the beacon 110, 112, the links to their manuals, or other maintenance information, for example. in this manner, the information displayed to the user may be reduced or expanded based on the proximity rules/actions associated with the asset.”
Schmidt discloses the following further limitation which DeYoung does not disclose:
identifying a target industrial asset within the plurality of target industrial assets based on the second set of images, wherein the target industrial asset is identified by determining that a degree of match between an image of the target industrial asset and at least one image from the plurality of images satisfies a predetermined threshold; and
[0035] “The diagnostic image linking system may then compare (e.g., using a distance metric) the feature vector to corresponding feature vectors of “ground truth” data for a collection of maintenance assets to generate similarity scores and to identify a specific maintenance asset within the collection of maintenance assets with the best match or greatest similarity to the machine captured by the set of machine-health diagnostic images.” (Emphasis added.)
[0036] “The diagnostic image linking system may generate a first similarity score using the first set of thermal images, a second similarity score using the second set of VL images, and a third similarity score using the third set of acoustic images. The diagnostic image linking system may compute a weighted overall similarity score using the first similarity score, the second similarity score, and the third similarity score.” (Emphasis added.)
At the time of filing, it would have been obvious to a person of ordinary skill to combine the techniques disclosed by Schmidt for identifying industrial assets using a similarity score with the DeYoung system because because—as suggested by Schmidt—industrial facilities may have many assets (machines) in close proximity. By identifying asset carefully (ie with high confidence), system technicians can ensure the usability of the monitoring system, and effectively proactively monitor this equipment.
Regarding claim 2, Schmidt discloses the further limitation wherein creating the first set of images comprises:
converting the location data to a first multi-dimensional dataset.
[0028] “The metadata information may include dates and times for when the sets of machine-health diagnostic images were captured and physical locations (e.g., a GPS location or an identification of a physical location within a building or other structure) corresponding with where the sets of machine-health diagnostic images were captured.”
The Examiner notes that GPS location data comprises at least latitude and longitude.
Regarding claim 3, DeYoung discloses the further limitation . The method of claim 1, wherein creating the second set of images comprises:
converting the process data to a second multi-dimensional dataset.
[0054] “Machine system 620 continually produces operational data over time. The operational data indicates the current status of machine system 620, such as parameters, pressure, temperature, speed, energy usage, operational equipment effectiveness (OEE), mean time between failure (MTBF), mean time to repair (MTTR), voltage, throughput volumes, times, tank levels, or any other performance status metrics. The operational data may comprise dynamic charts or trends, real-time video, or some other graphical content. Machine system 620 and/or controller 625 is capable of transferring the operational data over a communication link to database system 650, application integration platform 635, and computing system 610, typically via a communication network.”
Regarding claim 4, DeYoung discloses the further limitation comprising creating a third set of images, wherein the third set of images comprises the second set of images and the process data associated with the plurality of target industrial assets.
[0054] “Machine system 620 continually produces operational data over time. The operational data indicates the current status of machine system 620, such as parameters, pressure, temperature, speed, energy usage, operational equipment effectiveness (OEE), mean time between failure (MTBF), mean time to repair (MTTR), voltage, throughput volumes, times, tank levels, or any other performance status metrics. The operational data may comprise dynamic charts or trends, real-time video, or some other graphical content. Machine system 620 and/or controller 625 is capable of transferring the operational data over a communication link to database system 650, application integration platform 635, and computing system 610, typically via a communication network.”
Regarding claims 6 and 18, DeYoung discloses a method, (and a related portable device) comprising:
capturing a captured image of a candidate industrial asset on a portable device;
[0027] “Computing system 101 identifies (203) the asset to be associated with the beacon 110, 112. In one example, a camera of computing system is accessed and operated to scan a bar code 115-118 associated with the respective asset 105-108.”
in response to capturing the captured image, accessing a captured environmental input related to the candidate industrial asset;
[0020] “The techniques disclosed herein attempt to facilitate identification of industrial equipment by including micro-location devices with individual pieces of equipment and processing the identification signal transmitted by the micro-location devices to correlate the identifiers to their related assets.”
accessing a data record comprising indicia of the candidate industrial asset and at least one reference image of the candidate industrial asset accompanied by a reference environmental input; and …
Id.
‘environmental input’ :: [0027] ‘camera’.
in response to determining that the degree of match is above the predetermined threshold, enabling at least one management feature of the candidate industrial asset.
PNG
media_image1.png
478
256
media_image1.png
Greyscale
Fig. 5 (excerpt, reproduced above).
[0029] “Another action rule may be set up such when the computing system 101 is within a second, closer distance threshold (e.g., 5 feet) of the beacon, maintenance is likely being performed. In this case, the computing device 101 may be caused to automatically show the work orders for the devices associated with the beacon 110, 112, the links to their manuals, or other maintenance information, for example. in this manner, the information displayed to the user may be reduced or expanded based on the proximity rules/actions associated with the asset.”
Schmidt discloses the following further limitation which DeYoung does not disclose:
determining that a degree of match between the captured image and the at least one reference image and between the captured environmental input and the reference environmental input is above a predetermined threshold;
[0035] “The diagnostic image linking system may then compare (e.g., using a distance metric) the feature vector to corresponding feature vectors of “ground truth” data for a collection of maintenance assets to generate similarity scores and to identify a specific maintenance asset within the collection of maintenance assets with the best match or greatest similarity to the machine captured by the set of machine-health diagnostic images.” (Emphasis added.)
[0036] “The diagnostic image linking system may generate a first similarity score using the first set of thermal images, a second similarity score using the second set of VL images, and a third similarity score using the third set of acoustic images. The diagnostic image linking system may compute a weighted overall similarity score using the first similarity score, the second similarity score, and the third similarity score.” (Emphasis added.)
The obviousness analysis of claim 1 applies equally here. Regarding independent claim 18, the portable device ([0021] “In at least one implementation, a user operates a computing system such as a smartphone, tablet, or laptop”.) A processor and a memory (‘data storage’) are inherent in each of these disclosed devices.
Regarding claim 7, DeYoung discloses the further limitation comprising:
determining that either the degree of match between the captured image and the reference image or the degree of match between the captured environmental input and the reference environmental input is not above the predetermined threshold, blocking an operation associated with the at least one management feature of the candidate industrial asset.
[0030] “In another example, a command rule may be set up to prohibit command of the industrial asset 105-108 if the proximity is greater than a command proximity distance threshold (e.g., 8 feet). Accordingly, any user operating computing device 101 at a distance greater than the command proximity distance threshold will be denied control access to the asset. If the user reduces the proximity distance to a distance within the command proximity distance threshold, command of the industrial asset 105-108 may be possible.” (Emphasis added.)
Regarding claims 10 and 20, Schmidt discloses the further limitation wherein of the captured environmental input comprises at least one of a global positioning system (GPS) coordinate, a sensor reading, alarm data, equipment master data, and process data.
[0097] “In step 412, textual information is extracted from the first set of images. In some cases, the textual information may comprise overlay textual information that was added to the first set of images by a maintenance technician, textual information extracted from metadata associated with the first set of images, and/or textual information extracted from equipment labeling captured by the first set of images. In step 414, metadata information is extracted from the first set of images. The metadata information may comprise date and time information for when the first set of images were captured and location information (e.g., GPS location information) for where the first set of images were captured. In step 416, a location associated with the machine is determined using the metadata information.”
Regarding claim 12, Schmidt discloses the further limitation comprising:
retrieving historian data comprising historical readings associated with the candidate industrial asset; and
[0097] “In step 412, textual information is extracted from the first set of images. In some cases, the textual information may comprise overlay textual information that was added to the first set of images by a maintenance technician, textual information extracted from metadata associated with the first set of images, and/or textual information extracted from equipment labeling captured by the first set of images. In step 414, metadata information is extracted from the first set of images. The metadata information may comprise date and time information for when the first set of images were captured and location information (e.g., GPS location information) for where the first set of images were captured. In step 416, a location associated with the machine is determined using the metadata information.”
merging the historian data with the captured image.
Id.
Regarding claim 13, Schmidt discloses the further limitation comprising:
predicting a reading for the candidate industrial asset based on the historian data; and
[0053] “supervised machine learning”
[0056] “During a training phase, a machine learning model, such as one of the machine learning models 262, may be trained using the machine learning model trainer 260 to generate predicted answers using a set of labeled training data, such as training data 272. …. In at least one example, the training data set may correspond with historical data corresponding with a period of time (e.g., data generated over the past year or month) or image data corresponding with a particular type of capturing device (e.g., thermal images or VL images).” (Emphasis added.)
verifying the predicted reading for the candidate industrial asset matches an actual reading for the candidate industrial asset.
Id.
Regarding claim 14, Schmidt discloses the further limitation . The method of claim 6, wherein determining the degree of match between the captured image and the reference image comprises providing the captured image to a neural network trained to identify industrial assets from images and providing indica of the match therefrom.
[0004] “Machine learning techniques, such as application of deep learning neural network algorithms, may be used in parallel on image files, video files, and/or audiovisual files to perform image classification, object detection, object localization (e.g., to identify a location of an object within an image and/or determine a bounding box around the object), optical character recognition (OCR) text extraction, and audio speech-to-text conversions for audio recordings.”
Regarding claim 16, Schmidt discloses the further limitation comprising:
collecting at least one of process data and alarm data associated with an industrial site;
[0004] “In some cases, a diagnostic image linking system may acquire machine-health diagnostic images, apply object detection and other computer vision techniques to identify a particular machine using the machine-health diagnostic images, determine one or more machine properties for the particular machine using the machine-health diagnostic images,”
creating a training set based on the collected at least one of process data and alarm data associated with the industrial site;
[0078] “Machine learning techniques may be used to identify and extract the image features from the set of machine-health diagnostic images. The set of machine-health diagnostic images may be captured by a maintenance technician using a camera, such as the thermal camera 114 in FIG. 1 , during maintenance or repair of a maintenance asset in the field.“
training a neural network using the training set;
[0061] “neural network”
confirming that the neural network has been sufficiently trained using the training set; and
[0055] model testing.
providing the neural network, once sufficiently trained, with the captured environmental input for purposes of identifying the candidate industrial asset.
[0053] application of ML models to classification problems.
Regarding claim 19, DeYoung discloses the further limitation comprising a display device, wherein the processor is further enabled to:
display information describing a task associated with the at least one management feature of the industrial asset.
PNG
media_image1.png
478
256
media_image1.png
Greyscale
Fig. 5 (excerpt, reproduced above).
Claims 5, 8 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over DeYoung, Schmidt and Minisankar.
Regarding claim 5, Minisankar discloses the following further limitation which neither DeYoung/Schmidt discloses comprising determining one or more measurement readings associated with the target industrial asset based on the third set of images.
[0192] “In an example, in a fifth aspect supportive of increased accuracy, the system 100 may apply image brightening techniques described herein (e.g., adjust a brightness level) to an image in response to an unsuccessful measurement reading of a meter face. For example, if the server 110 is unable to detect lines at 365, identify a final pointer line at 375, or obtain a measurement device reading at 380, the server 110 may apply image adjust a brightness level (e.g., using histogram normalization) of an existing composite image, followed by repeating example operations associated with the first through fourth aspects above, which may support reading meter faces captured under relatively poor lighting conditions or poor image capture conditions (e.g., sunny/rainy weather, a bright background leading to meter face having poor exposure, poor camera quality, difficulty of capturing images of a meter face up close due to position of the device 105 with respect to the meter face, etc.).” (Emphasis added.)
At the time of filing, it would have been obvious to a person of ordinary skill to combine the technique disclosed by Minisankar for automatically reading equipment readings from images with the combined system of DeYoung/Schmidt because this would facilitate greater automation, ie it would facilitate continued operations while minimizing human efforts.
Regarding claim 8, DeYoung discloses the further limitation wherein the captured image comprises an artifact unique to a single industrial asset of all industrial assets at an industrial site.
[0042] “serial number”.
Regarding claim 11, Schmidt discloses the further limitation comprising:
retrieving, from a database, one or more historical environmental inputs associated with the candidate industrial asset;
[0056] “During a training phase, a machine learning model, such as one of the machine learning models 262, may be trained using the machine learning model trainer 260 to generate predicted answers using a set of labeled training data, such as training data 272. …. In at least one example, the training data set may correspond with historical data corresponding with a period of time (e.g., data generated over the past year or month) or image data corresponding with a particular type of capturing device (e.g., thermal images or VL images).” (Emphasis added.)
validating that the candidate industrial asset corresponds to a targeted industrial asset;
[0057] “The machine learning model trainer 260 may implement a machine learning algorithm that uses a training data set from the training data 272 to train the machine learning model and uses the evaluation data set to evaluate the predictive ability of the trained machine learning model. The predictive performance of the trained machine learning model may be determined by comparing predicted answers generated by the trained machine learning model with the target answers in the evaluation data set (or “ground truth” values).”
DeYoung discloses:
in response to validating that the candidate industrial asset corresponds to the targeted industrial asset, retrieving a task associated with the targeted industrial asset; and
PNG
media_image1.png
478
256
media_image1.png
Greyscale
Fig. 5 (excerpt reproduced above).
[0029] “Another action rule may be set up such when the computing system 101 is within a second, closer distance threshold (e.g., 5 feet) of the beacon, maintenance is likely being performed. In this case, the computing device 101 may be caused to automatically show the work orders for the devices associated with the beacon 110, 112, the links to their manuals, or other maintenance information, for example. in this manner, the information displayed to the user may be reduced or expanded based on the proximity rules/actions associated with the asset.”
Minisankar discloses the following further limitation which neither DeYoung/Schmidt disclose:
automatically filling in at least one measurement reading for the task associated with the targeted industrial asset.
[0192] “In an example, in a fifth aspect supportive of increased accuracy, the system 100 may apply image brightening techniques described herein (e.g., adjust a brightness level) to an image in response to an unsuccessful measurement reading of a meter face. For example, if the server 110 is unable to detect lines at 365, identify a final pointer line at 375, or obtain a measurement device reading at 380, the server 110 may apply image adjust a brightness level (e.g., using histogram normalization) of an existing composite image, followed by repeating example operations associated with the first through fourth aspects above, which may support reading meter faces captured under relatively poor lighting conditions or poor image capture conditions (e.g., sunny/rainy weather, a bright background leading to meter face having poor exposure, poor camera quality, difficulty of capturing images of a meter face up close due to position of the device 105 with respect to the meter face, etc.).” (Emphasis added.)
The obviousness analysis of claim 5 applies equally here.
Claims 9 is rejected under 35 U.S.C. 103 as being unpatentable over DeYoung, Schmidt and Xiong.
Regarding claim 9, Xiong discloses the following further limitation which DeYoung/Schmidt do not disclose wherein the captured environmental input comprises one or more acoustic values indicating at least one of volume, frequency, and variation in volume over a provided time unit, or variation in frequency over the provided time unit.
[0024] “The sound sensors may include sensors (e.g., a microphone) for ascertaining volume (e.g., decibel meter), frequency measurement, and/or distance (e.g., sonar, measuring time to echo)”
At the time of filing, it would have been obvious to a person of ordinary skill to combine the technique disclosed by Xiong for receiving and analyzing sound information with the combined system of DeYoung/Schmidt because this information can be useful in identifying particular machines, or in diagnosing potential mechanical problems with said machines.
Claim 15 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over DeYoung, Schmidt and Mikolajczyk.
Regarding claim 15, Schmidt discloses the further limitation comprising:
collecting a set of industrial asset images from a database;
[0097] “set of images”.
creating a second training set for a second stage of training comprising the first training set and digital non-industrial asset images that are incorrectly detected as industrial asset images after the first stage of training; and
See generally [0053]-[0055], describing supervised machine learning and model training/testing.
training the neural network in the second stage using the second training set.
[0053] “supervised machine learning” and applications to classification problems.
Mikolajczyk discloses the further limitation which neither DeYoung/Schmidt discloses comprising:
applying one or more transformations to each industrial asset image including mirroring, rotating, cropping, magnifying, demagnifying, translating, smoothing, or contrast reduction to create a modified set of digital industrial asset images;
P. 118, first col.: ‘Shear’, ‘Zoom in’, ‘Reflection’, ‘Rotation’.
creating a first training set comprising the collected set of digital industrial asset images, the modified set of digital industrial asset images, and a set of digital non-industrial asset images;
Id.
training the neural network in a first stage using the first training set;
P. 118, first col., “deep neural network model”.
At the time of filing, it would have been obvious to a person of ordinary skill to apply the data augmentation techniques disclosed by Mikolajczyk to the combined system of DeYoung/Schmidt because the former can improve image classification model performance.
Regarding claim 17, Schmidt discloses the further limitation comprising:
collecting a set of industrial asset sensor values from a database;
[0004] “The machine-health diagnostic images may include various types of images, such as thermal images, visible-light (VL) images, and/or acoustic images, captured using different imaging modalities (e.g., infrared sensor(s), VL sensor(s), and/or acoustic sensor(s).”
Mikolajczyk discloses the further limitation which neither DeYoung/Schmidt discloses comprising:
applying one or more transformations to each industrial asset sensor value in the set of industrial asset sensor values;
P. 118, first col.: ‘Shear’, ‘Zoom in’, ‘Reflection’, ‘Rotation’.
fusing the captured image with the one or more transformations to each industrial asset sensor value by changing pixel data in the captured image.
See generally p. 118, describing data (image) augmentation using transformations.
The obviousness analysis of claim 15 applies equally here.
Additional Relevant Prior Art
The following references were identified by the Examiner as being relevant to the disclosed invention, but are not relied upon in any particular prior art rejection:
Goldstein discloses a system for object identification system (for warehouse sites) featuring sensor fusion techniques for improved reliability. (US 11,829,945 B1)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Vincent Gonzales whose telephone number is (571) 270-3837. The examiner can normally be reached on Monday-Friday 7 a.m. to 4 p.m. MT. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang, can be reached at (571) 270-7092.
Information regarding the status of an application may be obtained from the USPTO Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov.
/Vincent Gonzales/Primary Examiner, Art Unit 2124