DETAILED ACTION
This action is written in response to the remarks and amendments dated 4/24/26. This action is made final. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
In view of the Applicant’s arguments—as well as the latest claim amendments—the Examiner withdraws all outstanding rejections under §101.
The Applicants argue that the previous art of record does not anticipate or render obvious the claims as currently amended. The Examiner provides updated prior art rejections below necessitated by the current amendments. Additional arguments are also addressed below.
Subject Matter Eligibility
In determining whether the claims are subject matter eligible, the examiner has considered and applied guidance from MPEP § 2106. The examiner finds that the independent claims are directed to the practical application of identifying industrial equipment (assets) based on image information. Furthermore, the combination of steps performed in the recited method cannot be practically performed as a mental process.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
The following are the references relied upon in the rejections below:
DeYoung (US 2020/0166904 A1)
Mikolajczyk (Mikolajczyk, Agnieszka, and Michał Grochowski. "Data augmentation for improving deep learning in image classification problem." In 2018 international interdisciplinary PhD workshop (IIPhDW), pp. 117-122. IEEE, 2018.)
Minisankar (US 2023/0057340 A1)
Ramasamy (Ramasamy, Rajkumar, and Vasuki Arumugam. "Digital watermarking—A tutorial." IEEE Potentials 41.4 (2022): 43-48.)
Schmidt (US 2023/0343066 A1)
Xiong (US 2022/0058591 A1, cited by Applicant in IDS dated 6/30/23.)
Claims 1, 4, 6-7, 10, 12-14, 16 and 18-23 are rejected under 35 U.S.C. 103 as being unpatentable over DeYoung, Schmidt and Ramasamy.
Regarding claim 1, DeYoung discloses a method of identifying tasks associated with an industrial asset, the method comprising:
obtaining, by the at least one processor, a plurality of images associated with the industrial asset;
[0027] “Computing system 101 identifies (203) the asset to be associated with the beacon 110, 112. In one example, a camera of computing system is accessed and operated to scan a bar code 115-118 associated with the respective asset 105-108.”
creating, by the at least one processor, a first set of images, wherein the first set of images comprises location data associated with the plurality of images;
[0020] “The techniques disclosed herein attempt to facilitate identification of industrial equipment by including micro-location devices with individual pieces of equipment and processing the identification signal transmitted by the micro-location devices to correlate the identifiers to their related assets.”
determining, by the at least one processor, a plurality of target industrial assets based on the first set of images in response to a first training model processing the first set of images;
[0027] “Computing system 101 identifies (203) the asset to be associated with the beacon 110, 112. In one example, a camera of computing system is accessed and operated to scan a bar code 115-118 associated with the respective asset 105-108. In an embodiment, the bar codes 115-118 may be designed to contain the identifying information for the associated asset 105-108. For example, the bar codes 115-118 may contain asset identification (ID) information, asset operation information, asset parameter information, and the like. In another embodiment, the bar codes 115-118 may provide an identifier information to computing system 101, which may then access an internal database or an external database (e.g., via communication link 121 and communication network 120 to an external database on application server 130) to acquire the information for the industrial asset 105-108 such as the information listed above.”
retrieving, by the at least one processor, secondary data associated with the plurality of target industrial assets;
[0029] “ For example, an action rule may be set up such that when the computing system 101 is within a first distance threshold (e.g., 20 feet) of the beacon, the computing system 101 is operated in a monitoring mode. Accordingly, this action rule may cause a monitoring module to be automatically shown on the computing device 101 when within the first distance threshold that shows the key data values for all the devices in the cabinet containing the beacon. Another action rule may be set up such when the computing system 101 is within a second, closer distance threshold (e.g., 5 feet) of the beacon, maintenance is likely being performed. In this case, the computing device 101 may be caused to automatically show the work orders for the devices associated with the beacon 110, 112, the links to their manuals, or other maintenance information, for example. in this manner, the information displayed to the user may be reduced or expanded based on the proximity rules/actions associated with the asset. More than one action may be configured for or within a given time or distance threshold.”
[0020] “In an industrial automation environment, investigation is primarily dependent on available plant-wide information, which is typically accessible over a network of machines and other equipment. For example, technicians and other users commonly review tag data, trends, alarms, documentation, incident reports, chat transcripts, screen captures, and other content that is generated during the course of operating an industrial enterprise. However, this information is often not readily accessible when interacting with a machine, control system, or some other asset of the industrial operation. Thus, maintaining inventory, identifying equipment, and tracking assets in an industrial manufacturing environment can be difficult with existing solutions. Even once a piece of equipment has been identified, it may be challenging to correlate the equipment to the related software, logical and physical topology, status information, security access, documentation, and other content associated with the equipment. The techniques disclosed herein attempt to facilitate identification of industrial equipment by including micro-location devices with individual pieces of equipment and processing the identification signal transmitted by the micro-location devices to correlate the identifiers to their related assets.” (Emphasis added.)
creating, by the at least one processor, a second set of images, wherein the second set of images comprises the secondary data and the first set of images; …
[0043] “ Examples of the information associated with asset 105-108 that could be retrieved by computing system 101 include operational data, machine data, images, screen graphics data, screen captures, video data, sound recordings, production processes, tag data, control information and logic, alarms, notifications, drive configurations, dashboards, human-machine interface (HMI) display screens, key performance indicators (KPIs), charts, trends, and other graphical content, simulation data, version numbers, catalogs, spare parts inventories, maintenance/repair schedules, links to documentation, electrical drawings, manuals, material safety data sheets, standard operating procedures, emergency procedures, cleanout procedures, reset procedures, safety procedures, incident reports including problems and solutions, chat transcripts, and any other information associated with the asset of the industrial environment.” (Emphasis added.)
based on identifying the target industrial asset, automatically retrieving, by the at least one processor, a task associated with the target industrial asset.
[0029] “Another action rule may be set up such when the computing system 101 is within a second, closer distance threshold (e.g., 5 feet) of the beacon, maintenance is likely being performed. In this case, the computing device 101 may be caused to automatically show the work orders for the devices associated with the beacon 110, 112, the links to their manuals, or other maintenance information, for example. in this manner, the information displayed to the user may be reduced or expanded based on the proximity rules/actions associated with the asset.”
Schmidt discloses the following further limitation which DeYoung does not disclose:
identifying, by the at least one processor, a target industrial asset within the plurality of target industrial assets in response to the second training model processing the second set of images,
[0035] “The diagnostic image linking system may then compare (e.g., using a distance metric) the feature vector to corresponding feature vectors of “ground truth” data for a collection of maintenance assets to generate similarity scores and to identify a specific maintenance asset within the collection of maintenance assets with the best match or greatest similarity to the machine captured by the set of machine-health diagnostic images.” (Emphasis added.)
wherein the target industrial asset is identified by determining that a degree of match between an image of the target industrial asset and at least one image from the plurality of images satisfies a predetermined threshold; and
Id.
Also [0036] “The diagnostic image linking system may generate a first similarity score using the first set of thermal images, a second similarity score using the second set of VL images, and a third similarity score using the third set of acoustic images. The diagnostic image linking system may compute a weighted overall similarity score using the first similarity score, the second similarity score, and the third similarity score.” (Emphasis added.)
At the time of filing, it would have been obvious to a person of ordinary skill to combine the techniques disclosed by Schmidt for identifying industrial assets using a similarity score with the DeYoung system because because—as suggested by Schmidt—industrial facilities may have many assets (machines) in close proximity. By identifying asset carefully (ie with high confidence), system technicians can ensure the usability of the monitoring system, and effectively proactively monitor this equipment.
Ramasamy discloses the following further limitation which neither DeYoung/Schmidt discloses:
wherein creating the second set of images comprises creating an array of pixel data that represents the secondary data and embedding the array of pixel data into the plurality of images by adding the array of pixel to the images to generate a modified image input for processing by a second training model,
P. 45, fig.4 (reproduced below), illustrating additive watermarking process.
PNG
media_image1.png
193
541
media_image1.png
Greyscale
At the time of filing, it would have been obvious to a person of ordinary skill to apply the additive watermarking technique disclosed by Ramasamy to the DeYoung/Schmidt system because this facilitates conveying information about a machine—eg “trends, alarms, documentation, incident reports, chat transcripts, screen captures, and other content that is generated during the course of operating” as disclosed by DeYoung at [0020]—along with an image of a particular machine.
Regarding claim 4, DeYoung discloses the further limitation comprising creating a third set of images, wherein the third set of images comprises the second set of images and the secondary data associated with the plurality of target industrial assets.
[0054] “Machine system 620 continually produces operational data over time. The operational data indicates the current status of machine system 620, such as parameters, pressure, temperature, speed, energy usage, operational equipment effectiveness (OEE), mean time between failure (MTBF), mean time to repair (MTTR), voltage, throughput volumes, times, tank levels, or any other performance status metrics. The operational data may comprise dynamic charts or trends, real-time video, or some other graphical content. Machine system 620 and/or controller 625 is capable of transferring the operational data over a communication link to database system 650, application integration platform 635, and computing system 610, typically via a communication network.”
Regarding claims 6 and 18, DeYoung discloses a method, (and a related portable device) comprising:
capturing a captured image of a candidate industrial asset on a portable device;
[0027] “Computing system 101 identifies (203) the asset to be associated with the beacon 110, 112. In one example, a camera of computing system is accessed and operated to scan a bar code 115-118 associated with the respective asset 105-108.”
collecting at least one process data and alarm data associated with an industrial asset;
[0020] “In an industrial automation environment, investigation is primarily dependent on available plant-wide information, which is typically accessible over a network of machines and other equipment. For example, technicians and other users commonly review tag data, trends, alarms, documentation, incident reports, chat transcripts, screen captures, and other content that is generated during the course of operating an industrial enterprise. However, this information is often not readily accessible when interacting with a machine, control system, or some other asset of the industrial operation. Thus, maintaining inventory, identifying equipment, and tracking assets in an industrial manufacturing environment can be difficult with existing solutions. Even once a piece of equipment has been identified, it may be challenging to correlate the equipment to the related software, logical and physical topology, status information, security access, documentation, and other content associated with the equipment. The techniques disclosed herein attempt to facilitate identification of industrial equipment by including micro-location devices with individual pieces of equipment and processing the identification signal transmitted by the micro-location devices to correlate the identifiers to their related assets.” (Emphasis added.)
creating a training set based on the collected at least one of process data and alarm data associated with the industrial asset; …
Id.
in response to capturing the captured image, providing the neural network, a captured environmental input that identifies the candidate industrial asset and accessing the captured environmental input related to the candidate industrial asset;
[0020] “The techniques disclosed herein attempt to facilitate identification of industrial equipment by including micro-location devices with individual pieces of equipment and processing the identification signal transmitted by the micro-location devices to correlate the identifiers to their related assets.”
accessing a data record comprising indicia of the candidate industrial asset and at least one reference image of the candidate industrial asset accompanied by a reference environmental input; and …
Id.
‘environmental input’ :: [0027] ‘camera’.
in response to determining that the degree of match is above the predetermined threshold, enabling at least one management feature including initiating a control operation that changes an operational state of the candidate industrial asset …
PNG
media_image2.png
478
256
media_image2.png
Greyscale
Fig. 5 (excerpt, reproduced above).
[0029] “Another action rule may be set up such when the computing system 101 is within a second, closer distance threshold (e.g., 5 feet) of the beacon, maintenance is likely being performed. In this case, the computing device 101 may be caused to automatically show the work orders for the devices associated with the beacon 110, 112, the links to their manuals, or other maintenance information, for example. in this manner, the information displayed to the user may be reduced or expanded based on the proximity rules/actions associated with the asset.”
wherein the training set is created by creating an array of pixel data that represents the collected at least one of process data and alarm data associated with the industrial asset …
[0020] “For example, technicians and other users commonly review tag data, trends, alarms, documentation, incident reports, chat transcripts, screen captures, and other content that is generated during the course of operating an industrial enterprise.”
Schmidt discloses the following further limitation which DeYoung does not disclose:
determining that a degree of match between the captured image and the at least one reference image and between the captured environmental input and the reference environmental input is above a predetermined threshold; …
[0035] “The diagnostic image linking system may then compare (e.g., using a distance metric) the feature vector to corresponding feature vectors of “ground truth” data for a collection of maintenance assets to generate similarity scores and to identify a specific maintenance asset within the collection of maintenance assets with the best match or greatest similarity to the machine captured by the set of machine-health diagnostic images.” (Emphasis added.)
[0036] “The diagnostic image linking system may generate a first similarity score using the first set of thermal images, a second similarity score using the second set of VL images, and a third similarity score using the third set of acoustic images. The diagnostic image linking system may compute a weighted overall similarity score using the first similarity score, the second similarity score, and the third similarity score.” (Emphasis added.)
training a neural network using the training set;
[0050] “The one or more machine learning models may include neural networks ( e.g., deep neural networks), …. A neural network may comprise a feedforward neural network, recurrent neural network, or a convolutional neural network.”
The obviousness analysis of claim 1 applies equally here. Regarding independent claim 18, the portable device ([0021] “In at least one implementation, a user operates a computing system such as a smartphone, tablet, or laptop”.) A processor and a memory (‘data storage’) are inherent in each of these disclosed devices.
Ramasamy discloses the following further limitation which neither DeWitt/Schmidt discloses:
… embedding the array of pixel data into a plurality of images associated with the candidate industrial asset.
P. 45, fig.4 (reproduced below), illustrating additive watermarking process.
PNG
media_image3.png
193
541
media_image3.png
Greyscale
The obviousness analysis of claim 1 applies equally here.
Regarding claim 7, DeYoung discloses the further limitation comprising:
determining that either the degree of match between the captured image and the reference image or the degree of match between the captured environmental input and the reference environmental input is not above the predetermined threshold, blocking an operation associated with the at least one management feature of the candidate industrial asset.
[0030] “In another example, a command rule may be set up to prohibit command of the industrial asset 105-108 if the proximity is greater than a command proximity distance threshold (e.g., 8 feet). Accordingly, any user operating computing device 101 at a distance greater than the command proximity distance threshold will be denied control access to the asset. If the user reduces the proximity distance to a distance within the command proximity distance threshold, command of the industrial asset 105-108 may be possible.” (Emphasis added.)
Regarding claims 10 and 20, Schmidt discloses the further limitation wherein of the captured environmental input comprises at least one of a global positioning system (GPS) coordinate, a sensor reading, alarm data, equipment master data, and process data.
[0097] “In step 412, textual information is extracted from the first set of images. In some cases, the textual information may comprise overlay textual information that was added to the first set of images by a maintenance technician, textual information extracted from metadata associated with the first set of images, and/or textual information extracted from equipment labeling captured by the first set of images. In step 414, metadata information is extracted from the first set of images. The metadata information may comprise date and time information for when the first set of images were captured and location information (e.g., GPS location information) for where the first set of images were captured. In step 416, a location associated with the machine is determined using the metadata information.”
Regarding claim 12, Schmidt discloses the further limitation comprising:
retrieving historian data comprising historical readings associated with the candidate industrial asset; and
[0097] “In step 412, textual information is extracted from the first set of images. In some cases, the textual information may comprise overlay textual information that was added to the first set of images by a maintenance technician, textual information extracted from metadata associated with the first set of images, and/or textual information extracted from equipment labeling captured by the first set of images. In step 414, metadata information is extracted from the first set of images. The metadata information may comprise date and time information for when the first set of images were captured and location information (e.g., GPS location information) for where the first set of images were captured. In step 416, a location associated with the machine is determined using the metadata information.”
merging the historian data with the captured image.
Id.
Regarding claim 13, Schmidt discloses the further limitation comprising:
predicting a reading for the candidate industrial asset based on the historian data; and
[0053] “supervised machine learning”
[0056] “During a training phase, a machine learning model, such as one of the machine learning models 262, may be trained using the machine learning model trainer 260 to generate predicted answers using a set of labeled training data, such as training data 272. …. In at least one example, the training data set may correspond with historical data corresponding with a period of time (e.g., data generated over the past year or month) or image data corresponding with a particular type of capturing device (e.g., thermal images or VL images).” (Emphasis added.)
verifying the predicted reading for the candidate industrial asset matches an actual reading for the candidate industrial asset.
Id.
Regarding claim 14, Schmidt discloses the further limitation . The method of claim 6, wherein determining the degree of match between the captured image and the reference image comprises providing the captured image to a neural network trained to identify industrial assets from images and providing indica of the match therefrom.
[0004] “Machine learning techniques, such as application of deep learning neural network algorithms, may be used in parallel on image files, video files, and/or audiovisual files to perform image classification, object detection, object localization (e.g., to identify a location of an object within an image and/or determine a bounding box around the object), optical character recognition (OCR) text extraction, and audio speech-to-text conversions for audio recordings.”
Regarding claim 16, Schmidt discloses the further limitation comprising:
collecting at least one of process data and alarm data associated with an industrial site;
[0004] “In some cases, a diagnostic image linking system may acquire machine-health diagnostic images, apply object detection and other computer vision techniques to identify a particular machine using the machine-health diagnostic images, determine one or more machine properties for the particular machine using the machine-health diagnostic images,”
creating a training set based on the collected at least one of process data and alarm data associated with the industrial site;
[0078] “Machine learning techniques may be used to identify and extract the image features from the set of machine-health diagnostic images. The set of machine-health diagnostic images may be captured by a maintenance technician using a camera, such as the thermal camera 114 in FIG. 1 , during maintenance or repair of a maintenance asset in the field.“
training a neural network using the training set;
[0061] “neural network”
confirming that the neural network has been sufficiently trained using the training set; and
[0055] model testing.
providing the neural network, once sufficiently trained, with the captured environmental input for purposes of identifying the candidate industrial asset.
[0053] application of ML models to classification problems.
Regarding claim 19, DeYoung discloses the further limitation comprising a display device, wherein the processor is further enabled to:
display information describing a task associated with the at least one management feature of the industrial asset.
PNG
media_image2.png
478
256
media_image2.png
Greyscale
Fig. 5 (excerpt, reproduced above).
Regarding claim 21, Schmidt discloses the further limitations wherein determining the plurality of target industrial assets based on the first set of images in response to the first training model processing the first set of images comprises:
identifying, by the at least one processor, a list of target industrial assets from a master equipment list of all industrial assets at an industrial site; and
[0063] “In some cases, the asset profiles database 232 may store machine-specific data for a collection of maintenance assets. The machine-specific data may include the dimensions of a machine or equipment, a temperature of the machine or equipment, locations of various external and internal features of the machine ( e.g., locations of various dials and buttons on the outside of the machine and the types of components and their locations inside the machine) or equipment”. (Emphasis added.)
[0097] “The metadata information may comprise date and time information for when the first set of images were captured and location information (e.g., GPS location information) for where the first set of images were captured. In step 416, a location associated with the machine is determined using the metadata information.”
filtering, by the at least one processor, the master equipment list to include only those industrial assets that are within a previously determined distance from a location.
[0037] “In at least one example, the weighted coefficient for the first similarity score may be reduced (e.g., cut in half) if the estimated distance from the thermal camera to the machine is greater than a threshold distance (e.g., is greater than ten feet). In another example, the weighted coefficient for the second similarity score may be set based on the estimated distance from the VL camera to the machine.” (Emphasis added.)
Regarding claim 22, DeYoung discloses the further limitation wherein the second training model is trained based on the second set of images, to identify the target industrial asset within the plurality of target industrial assets based on an indicia associated with the secondary data, wherein the indicia is implicit.
[0020] “In an industrial automation environment, investigation is primarily dependent on available plant-wide information, which is typically accessible over a network of machines and other equipment. For example, technicians and other users commonly review tag data, trends, alarms, documentation, incident reports, chat transcripts, screen captures, and other content that is generated during the course of operating an industrial enterprise. However, this information is often not readily accessible when interacting with a machine, control system, or some other asset of the industrial operation. Thus, maintaining inventory, identifying equipment, and tracking assets in an industrial manufacturing environment can be difficult with existing solutions. Even once a piece of equipment has been identified, it may be challenging to correlate the equipment to the related software, logical and physical topology, status information, security access, documentation, and other content associated with the equipment. The techniques disclosed herein attempt to facilitate identification of industrial equipment by including micro-location devices with individual pieces of equipment and processing the identification signal transmitted by the micro-location devices to correlate the identifiers to their related assets.” (Emphasis added.)
Regarding claim 23, Ramasamy discloses the further limitation wherein the collected at least one of process data and alarm data associated with the industrial asset is converted to create an array of pixel data, and the array of pixel data is added into the captured image of a candidate industrial asset for training the neural network.
P. 45, fig.4 (reproduced supra), illustrating additive watermarking process.
Training a neural network for subsequent equipment identification is disclosed by Schmidt; see mapping for claim 1 supra.
Claims 5, 8 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over DeYoung, Schmidt, Ramasamy and Minisankar.
Regarding claim 5, Minisankar discloses the following further limitation which neither DeYoung/Schmidt/Ramasamy discloses comprising determining one or more measurement readings associated with the target industrial asset based on the third set of images.
[0192] “In an example, in a fifth aspect supportive of increased accuracy, the system 100 may apply image brightening techniques described herein (e.g., adjust a brightness level) to an image in response to an unsuccessful measurement reading of a meter face. For example, if the server 110 is unable to detect lines at 365, identify a final pointer line at 375, or obtain a measurement device reading at 380, the server 110 may apply image adjust a brightness level (e.g., using histogram normalization) of an existing composite image, followed by repeating example operations associated with the first through fourth aspects above, which may support reading meter faces captured under relatively poor lighting conditions or poor image capture conditions (e.g., sunny/rainy weather, a bright background leading to meter face having poor exposure, poor camera quality, difficulty of capturing images of a meter face up close due to position of the device 105 with respect to the meter face, etc.).” (Emphasis added.)
At the time of filing, it would have been obvious to a person of ordinary skill to combine the technique disclosed by Minisankar for automatically reading equipment readings from images with the combined system of DeYoung/Schmidt/Ramasamy because this would facilitate greater automation, ie it would facilitate continued operations while minimizing human efforts.
Regarding claim 8, DeYoung discloses the further limitation wherein the captured image comprises an artifact unique to a single industrial asset of all industrial assets at an industrial site.
[0042] “serial number”.
Regarding claim 11, Schmidt discloses the further limitation comprising:
retrieving, from a database, one or more historical environmental inputs associated with the candidate industrial asset;
[0056] “During a training phase, a machine learning model, such as one of the machine learning models 262, may be trained using the machine learning model trainer 260 to generate predicted answers using a set of labeled training data, such as training data 272. …. In at least one example, the training data set may correspond with historical data corresponding with a period of time (e.g., data generated over the past year or month) or image data corresponding with a particular type of capturing device (e.g., thermal images or VL images).” (Emphasis added.)
validating that the candidate industrial asset corresponds to a targeted industrial asset;
[0057] “The machine learning model trainer 260 may implement a machine learning algorithm that uses a training data set from the training data 272 to train the machine learning model and uses the evaluation data set to evaluate the predictive ability of the trained machine learning model. The predictive performance of the trained machine learning model may be determined by comparing predicted answers generated by the trained machine learning model with the target answers in the evaluation data set (or “ground truth” values).”
DeYoung discloses:
in response to validating that the candidate industrial asset corresponds to the targeted industrial asset, retrieving a task associated with the targeted industrial asset; and
PNG
media_image2.png
478
256
media_image2.png
Greyscale
Fig. 5 (excerpt reproduced above).
[0029] “Another action rule may be set up such when the computing system 101 is within a second, closer distance threshold (e.g., 5 feet) of the beacon, maintenance is likely being performed. In this case, the computing device 101 may be caused to automatically show the work orders for the devices associated with the beacon 110, 112, the links to their manuals, or other maintenance information, for example. in this manner, the information displayed to the user may be reduced or expanded based on the proximity rules/actions associated with the asset.”
Minisankar discloses the following further limitation which neither DeYoung/Schmidt/Ramasamy discloses:
automatically filling in at least one measurement reading for the task associated with the targeted industrial asset.
[0192] “In an example, in a fifth aspect supportive of increased accuracy, the system 100 may apply image brightening techniques described herein (e.g., adjust a brightness level) to an image in response to an unsuccessful measurement reading of a meter face. For example, if the server 110 is unable to detect lines at 365, identify a final pointer line at 375, or obtain a measurement device reading at 380, the server 110 may apply image adjust a brightness level (e.g., using histogram normalization) of an existing composite image, followed by repeating example operations associated with the first through fourth aspects above, which may support reading meter faces captured under relatively poor lighting conditions or poor image capture conditions (e.g., sunny/rainy weather, a bright background leading to meter face having poor exposure, poor camera quality, difficulty of capturing images of a meter face up close due to position of the device 105 with respect to the meter face, etc.).” (Emphasis added.)
The obviousness analysis of claim 5 applies equally here.
Claims 9 is rejected under 35 U.S.C. 103 as being unpatentable over DeYoung, Schmidt, Ramasamy and Xiong.
Regarding claim 9, Xiong discloses the following further limitation which DeYoung/Schmidt/Ramasamy do not disclose wherein the captured environmental input comprises one or more acoustic values indicating at least one of volume, frequency, and variation in volume over a provided time unit, or variation in frequency over the provided time unit.
[0024] “The sound sensors may include sensors (e.g., a microphone) for ascertaining volume (e.g., decibel meter), frequency measurement, and/or distance (e.g., sonar, measuring time to echo)”
At the time of filing, it would have been obvious to a person of ordinary skill to combine the technique disclosed by Xiong for receiving and analyzing sound information with the combined system of DeYoung/Schmidt/Ramasamy because this information can be useful in identifying particular machines, or in diagnosing potential mechanical problems with said machines.
Claim 15 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over DeYoung, Schmidt, Ramasamy and Mikolajczyk.
Regarding claim 15, Schmidt discloses the further limitation comprising:
collecting a set of industrial asset images from a database;
[0097] “set of images”.
creating a second training set for a second stage of training comprising the first training set and digital non-industrial asset images that are incorrectly detected as industrial asset images after the first stage of training; and
See generally [0053]-[0055], describing supervised machine learning and model training/testing.
training the neural network in the second stage using the second training set.
[0053] “supervised machine learning” and applications to classification problems.
Mikolajczyk discloses the further limitation which neither DeYoung/Schmidt/Ramasamy discloses comprising:
applying one or more transformations to each industrial asset image including mirroring, rotating, cropping, magnifying, demagnifying, translating, smoothing, or contrast reduction to create a modified set of digital industrial asset images;
P. 118, first col.: ‘Shear’, ‘Zoom in’, ‘Reflection’, ‘Rotation’.
creating a first training set comprising the collected set of digital industrial asset images, the modified set of digital industrial asset images, and a set of digital non-industrial asset images;
Id.
training the neural network in a first stage using the first training set;
P. 118, first col., “deep neural network model”.
At the time of filing, it would have been obvious to a person of ordinary skill to apply the data augmentation techniques disclosed by Mikolajczyk to the combined system of DeYoung/Schmidt/Ramasamy because the former can improve image classification model performance.
Regarding claim 17, Schmidt discloses the further limitation comprising:
collecting a set of industrial asset sensor values from a database;
[0004] “The machine-health diagnostic images may include various types of images, such as thermal images, visible-light (VL) images, and/or acoustic images, captured using different imaging modalities (e.g., infrared sensor(s), VL sensor(s), and/or acoustic sensor(s).”
Mikolajczyk discloses the further limitation which neither DeYoung/Schmidt/Ramasamy discloses comprising:
applying one or more transformations to each industrial asset sensor value in the set of industrial asset sensor values;
P. 118, first col.: ‘Shear’, ‘Zoom in’, ‘Reflection’, ‘Rotation’.
fusing the captured image with the one or more transformations to each industrial asset sensor value by changing pixel data in the captured image.
See generally p. 118, describing data (image) augmentation using transformations.
The obviousness analysis of claim 15 applies equally here.
Additional Relevant Prior Art
The following references were identified by the Examiner as being relevant to the disclosed invention, but are not relied upon in any particular prior art rejection:
Goldstein discloses a system for object identification system (for warehouse sites) featuring sensor fusion techniques for improved reliability. (US 11,829,945 B1)
Kar discloses a system for encoding metadata about equipment using image watermarks. (See [0048] “In one embodiment, the information identifying the equipment may be included in a metadata file associated with the received one or more images 306 or documents 308.”) (US 2020/0151784 A1)
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Vincent Gonzales whose telephone number is (571) 270-3837. The examiner can normally be reached on Monday-Friday 7 a.m. to 4 p.m. MT. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Miranda Huang, can be reached at (571) 270-7092.
Information regarding the status of an application may be obtained from the USPTO Patent Center.
/Vincent Gonzales/Primary Examiner, Art Unit 2124
Prosecution Insights