DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/21/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
The information disclosure statement (IDS) submitted on 07/17/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
The information disclosure statement (IDS) submitted on 08/19/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
The information disclosure statement (IDS) submitted on 02/06/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Response to Preliminary Amendment
The preliminary amendment filed on 05/21/2024 have been acknowledged.
Claims 1, 2, 3, 14, 20, 23, 24, 33, 41, 43, 44, 46, 54, 57, 64, 66, 67, 76, 77, 84, are original and are pending.
Claims 4-13, 15-19, 21, 22, 25-32, 34-40, 42, 45, 47-53, 55, 56, 58-63, 65, 68-75, 78-83 are cancelled.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: “artificial intelligence (AI) module” , and “analysis module“, in claim 1 as well as claim 1’s dependent claims, “artificial intelligence (AI) module” in claim 41 and its dependent claims , “recalibration module” in claim 77 and “artificial intelligence (AI) module” in claim 84.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim 1, 2, 3, 41, 46, 54, 64 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Benouis et al (Benouis et al hereinafter, Food Tray Sealing Fault Detection in Multi-Spectral Images Using Data Fusion and Deep Learning Techniques).
As per claim 1
Benouis teaches A system comprising an imager configured to acquire images of a sample ( 2.1Dataset: “ The hyperspectral camera of the HSI system used to acquire the images was a Pika NIR-320 “), an artificial intelligence (AI) module, trained to identify, within multi-dimensional image data corresponding to images of objects, wavelength patterns corresponding to one or more defects within the objects (Benouis explains that hyperspectral imaging produces a multi dimension data cube containing spatial and spectral images: “For each image frame, data cubes generated by HSI sensors contain 3D data (𝑊×𝐻×𝐷) corresponding to an image in size of W pixels in width, H pixels in height, and D spectral channels in depth, i.e., two dimensions in spatial information and one dimension in spectral information, usually called data cube.” Benouis tells us that the “data cube contained 130-channels, from 984.2 to 1631.0 (nm).” Benouis then gives disclosure that the wavelengths correspond to a certain defect. : “. Figure 2 shows the spectral signatures of some of the anomalies: washer, sugar, cork, and polarized plastic”. Benouis also discloses training machine learning models to detect the anomalies. Introduction: “The proposed machine learning algorithm detects any anomaly located in the seal by analyzing the mean value of spectral bands in the data cube” Benouis discloses that four different classifiers and used with the input data. The architecture is shown in figure 1. This includes a DBN, ELM, SAE and CNN. Since the machine learning algorithm analyzes spectral bands in the hyperspectral data cube and uses spectral signatures of anomalies (defects)they are effectively trained to identify wavelength dependent patterns which correlate to defects.) and an analysis module configured to detect using the AI module, one or more defects in the sample (Benouis’ machine learning system is used to detect defects “The proposed machine learning algorithm detects any anomaly located in the seal”. The deep learning classifier processes hyperspectral data to determine condition: Abstract “checking a food tray sealing state (faulty or normal),” This shows that the trained machine learning classifier that processes hyperspectral data to determine whether the target is faulty or normal corresponds directly to the claimed “analysis module configured to detect defects using the AI module”).
As per claim 2
Benouis shows an analysis pipeline that classifies defect state into faulty or normal using ML/DL classifiers and maps defects via ROI around a contamination area. Benouis teaches the analysis module is configured to classify detected defect (Abstract “A deep learning-based approach for food tray sealing fault detection using hyperspectral images is described. Several pixel-based image fusion methods are proposed to obtain 2D images from the 3D hyperspectral image data cube, which feeds the deep learning (DL) algorithms…while maintaining a high classification ratio, showing that the fused image contains enough information for checking a food tray sealing state (faulty or normal)” Benouis also provides a labeled dataset that the classifier uses: Dataset “For each food tray, multiple non-overlapping Regions Of Interest (ROI) were extracted. Thus, a dataset was obtained, with 2112 valid ROIs from which 1123 were labeled as “normal” and 989 as “faulty”.” As stated in claim 1’s 103 rejection, Benouis shows use of machine learning classifiers in that task (figure 1). Essentially Benouis’ system checks the state of faulty vs normal using ML classifiers.) and or map the defect (The word “map” can be interpreted to simply localize it in a sample. Benouis teaches spatial localization using ROIs around contamination. Dataset: “Manual processing was made on the dataset by applying ROI filtering around a contamination area “ and then shows defect locality and treats regions distinctly “. Figure 2 shows the spectral signatures of some of the anomalies: washer, sugar, cork, and polarized plastic. For each of them, the curves represent the anomaly region (green area), the edge region without anomaly (tray-edge label, orange area), and an inner region of the tray (tray-center label, red area); the normal edge region and the tray center region are included as a comparison with the faulty region.”).
As per claim 3
Benouis explains a hyperspectral imaging creates a 3D data cube consisting of 2 spatial dimension and 1 spectral dimension which inherently means the imager is a “multi-dimensional” imager (Introduction: “For each image frame, data cubes generated by HSI sensors contain 3D data (𝑊×𝐻×𝐷) corresponding to an image in size of W pixels in width, H pixels in height, and D spectral channels in depth, i.e., two dimensions in spatial information and one dimension in spectral information, usually called data cube. Then, 4D hyper cubes are generated if we consider time as the fourth dimension.”).
As per claim 41
Claim 41 is the method claim that parallels claim 1’s system claim and are therefore rejected under the same reasoning.
As per claim 46
Benouis covers all claim limitations rejected in claim 41’s 102 rejection. See claim 41’s 102 rejection.
Benouis teaches acquisition of images of a sample comprises collecting multi-dimensional data of the sample using a multi-dimensional imager (“For each image frame, data cubes generated by HSI sensors contain 3D data (𝑊×𝐻×𝐷) corresponding to an image in size of W pixels in width, H pixels in height, and D spectral channels in depth, i.e., two dimensions in spatial information and one dimension in spectral information, usually called data cube” and “A line scan camera composed of 410 lines by 320 pixels with 168 channels (wavelengths in the range from 891.1 to 1728.4 nm) was used to scan the food packaging. The hyperspectral camera of the HSI system used to acquire the images was a Pika NIR-320 “).
As per claim 54
Benouis teaches the artificial intelligence module is trained with a training set of multi-dimensional images processed to classify spatio-spectral signatures for the defects. (Figure 1, Results section : “Once the HSI image had been transformed into a 2D single image through fusion techniques, we applied deep learning techniques to train the transformed HSI images (with and without anomaly). The training followed a five-fold cross-validation procedure.” Data fusion techniques section: “In order to reduce the extra computation burden and the “Hughes phenomenon” typically appearing during DL training, the dimension of the hyper-spectral images was transformed using different fusion methods. Figure 3 shows the data fusion procedures proposed in this work” and Figure 3).
As per claim 64
Benouis teaches the step of reducing a number of dimensions of the multi-dimensional data (Data fusion techniques “Let 𝐼(𝑗,𝑗,𝐿) be a data cube containing the hyperspectral image, where M is the number of rows, N the number of columns, and L is the number of spectral bands, i.e., an 𝑀×𝑁 pixel array with L spectral channels for each pixel. The proposed data fusion procedures convert this data cube into a 2D data matrix 𝐹(𝑖,𝑗), which can be associated with an image, i.e., the hyperspectral image (3D) is transformed in a 2D data matrix (image).”).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 14 and 57, are rejected under 35 U.S.C. 103 as being unpatentable over Benouis et al (Benouis et al hereinafter, Food Tray Sealing Fault Detection in Multi-Spectral Images Using Data Fusion and Deep Learning Techniques) in view of Rao et al (Rao hereinafter Quaternion Based Neural Network for Hyperspectral Image Classification).
Benouis covers all the claim limitations of claim 1’s 102 rejection.
Benouis does not describe a hypercomplex neural network within their machine learning/deep learning architecture.
Rao teaches using a quaternion neural network for hyperspectral image analysis (Introduction: “a novel quaternion hyperspectral image classification network (QHIC Net), which represents the hyperspectral data in the quaternion domain, is proposed.”)
Accordingly at the time this invention was effectively filed a person of ordinary skill in the art would have been motivated to incorporate the quaternion neural network of Rao in addition to the neural network classifiers disclosed by Benouis within the hyperspectral defect detection pipeline. Benouis already showcases using neural network classifiers to analyze spectral info within the hyperspectral data cube to detect anomalies. Rao shows that quaternion neural networks are particularly suited for hyperspectral image analysis because they can capture relationships between spectral channels. Rao even goes further and speculates that “the possibility of extending the quaternion hyperspectral space to tackle other issues such as hyperspectral band selection, unmixing, target and anomaly detection” . A person of ordinary skill in the art understands that Rao’s quaternion neural network could be added as an additional processing stage within the hyperspectral analysis pipeline. It could be used to process hyperspectral features prior to classification or to refine classification results by produced by initial neural network. That they can be used in tandem or in series. A person of ordinary skill in the art knows the quaternion neural network would operate in cascade with the existing classifiers enabling the system (Ai module) to leverage the perks of quaternion domain processing of spectral channels while retaining the architecture of Benouis. A person of ordinary skill in the art would know the modification gives the system the ability to capture inter channel spectral relations within the data improving representation of feature and plausibly increase anomaly detection.
As per claim 57
Benouis covers all claim limitations previously rejected in claim 41’s 102 rejection. Please see claim 41’s 102 rejection
Claim 57 is the method claim of claim 14 and will be rejected under the same premise.
Claim 20, 23, 24, 66, 67 and 84 are rejected under 35 U.S.C. 103 as being unpatentable over Benouis et al (Benouis et al hereinafter, Food Tray Sealing Fault Detection in Multi-Spectral Images Using Data Fusion and Deep Learning Techniques) in view of Radhakrishnan et al (Radhakrishnan hereinafter US 20210106231 A1).
As per claim 20
Benouis covers all claim limitations previously rejected in claim 1’s 102 rejection. Please see claim 1’s 102 rejection. Benouis provides a compatible spectral pattern defect detection framework.
Benouis teaches the defect is contamination of the seal but does not explicitly provide what that contamination actually is. Benouis doesn’t explicitly say the defects (contamination) comprise pathogens and the wavelength patterns each correspond to a specific pathogen.
Radhakrishnan teaches the defects detected comprises pathogens (Paragraph [0083] “A handheld device was fabricated for in-situ detection and classification of pathogens.”) and the wavelength patterns each correspond to a particular pathogen (Paragraph [0034] “clinically relevant pathogens may be distinguished based on their signature/characteristic spectral parameters extracted from their time-resolved fluorescence spectra and time-resolved reflectance, and time-resolved transmittance spectra, at various designated spectral band” and “The image processing module 104 also comprises a library database 118 comprising a set of standard spectral parameters identifiable with various reference pathogens.” The claimed “wavelength patterns” are Radhakrishnan’s “ multiple spectral bands “ and in paragraph [0080] they say “h of the plurality of spectral parameters is compared with the set of standard spectral parameters to detect and classify the pathogens”.)
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have modified Benouis’ pipeline with Radhakrishnan’s concept of detecting pathogen using the wavelength patterns that correspond to that particular pathogen. Both Benouis and Radhakrishnan both address the same technical flow. This is multi (hyper) spectral imagining of a physical sample followed by automated defect detection using spectral band information. Radhakrishnan teaches a predictable feature capability that Radhakrishnan does not; identifying the contamination (defect) as a particular pathogen by matching measured spectral parameters to a pathogen. Benouis already teaches detecting a defect anomaly by “analyzing the mean value of spectral bands in the data cube” while Radhakrishnan teaches acquiring spectra over a “predetermined range of wavelengths” capturing spectra at “multiple spectra bands” and comparing measured spectral parameters to “standard spectral parameters identifiable with reference pathogens” including identifying “a specific reference pathogen” In view of this it would have been obvious for a person of ordinary skill in the art to implement the pathogen identification logic as an added stage within Benouis’ existing hyperspectral anomaly detection workflow rather than replacing the deep learning architecture. The architecture’s first stage flags a region or sample as defective based on spectral band behavior in the data cube and the newly added second stage applies Radhakrishnan pathogen matching to the same spectral band derived parameters to determine the defect corresponds to a particular pathogen or another. The integration is straightforward and lacks impedance because both references rely on multi wavelength spectral information as a discriminating signal. The combination enables the benefit of Benouis’ system to not only detect anomalies in regards to contamination but also attributes a detected biological factor in regards to that contamination; a specified pathogen. All this while still leveraging the same spectral band inputs already processed.
As for claim 23
Benouis and Radhakrishnan cover the limitations previously rejected in claim 20 which subsequently includes claim 1.
Benouis teaches that the sample comprises food (Abstract :“A correct food tray sealing is required to preserve food properties and safety “ then Benouis continues in his Introduction stating “Thus, packaging technology is considered a vital step to ensure the quality of the food and prevent human poisoning.” Then later states “In our case, our first contribution is a novel hyperspectral food inspection algorithm based on a PCANet network [27]. The proposed machine learning algorithm detects any anomaly located in the seal by analyzing the mean value of spectral bands in the datacube” This is how we know it isn’t just an abiotic food tray being that comprises the sample.)
As previously stated Benouis does not provide what the faulty/defect classification (contamination) actually is. Therefore, Benouis does not explicitly teach that the defect being classified is particular pathogen.
Radhakrishnan teaches that the sample comprises food ( Paragraph [0070]“The clinically relevant pathogens that may be detected and classified using the device 100, of the present invention include, but are not limited to …in case of food contamination, Salmonella typhi may be of relevance.” Then in paragraph 0071 discloses that “The device 100 of the present invention is suitable for studying the pathogens present in various kinds of samples 112. The sample may comprise one or more of the following: …a consumable commodity…” ). Radhakrishnan teaches the issue detected comprises pathogens (Paragraph [0083] “A handheld device was fabricated for in-situ detection and classification of pathogens.”) and the wavelength patterns each correspond to a particular pathogen (Paragraph [0034] “clinically relevant pathogens may be distinguished based on their signature/characteristic spectral parameters extracted from their time-resolved fluorescence spectra and time-resolved reflectance, and time-resolved transmittance spectra, at various designated spectral band” and “The image processing module 104 also comprises a library database 118 comprising a set of standard spectral parameters identifiable with various reference pathogens.” The claimed “wavelength patterns” are Radhakrishnan’s “ multiple spectral bands “ and in paragraph [0080] they say “h of the plurality of spectral parameters is compared with the set of standard spectral parameters to detect and classify the pathogens”.)
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to further extend Benouis’ concept of multi/hyperspectral AI based inspection system for food tray sealing fault/contamination detection (i.e. defect detection in food related sample) to include pathogen type contamination defects. The extension can be done by incorporating Radhakrishnan’s pathogen detection. Both references rely on analyzing multi wavelength spectral information to detect abnormal conditions. Adding pathogen detection to classify pathogen as the exact contamination causing the faulty classification is a predictable sensing and analysis pipeline which can be used to broaden the set of detectable defects in food safety inspection contexts.
As for claim 24
Benouis and Radhakrishnan cover the limitations previously rejected in claim 20 which subsequently includes claim 1.
Benouis teaches the sample is an abiotic object (Dataset: “A line scan camera composed of 410 lines by 320 pixels with 168 channels (wavelengths in the range from 891.1 to 1728.4 nm) was used to scan the food packaging. “ And Abstract: “ These techniques greatly improve the computation time while maintaining a high classification ratio, showing that the fused image contains enough information for checking a food tray sealing state (faulty or normal),” The packaging itself comprises food but also the seal itself is abiotic.)
Radhakrishnan teaches the issue detected and classified in samples comprises pathogens (Paragraph [0083] “A handheld device was fabricated for in-situ detection and classification of pathogens.” and Paragraph [0076] “a sample comprising a plurality of pathogens is exposed “) and that detection sampling can be done on an abiotic surfaces (Paragraph [0039] “The device may be configured to study effectiveness of disinfectants on various hospital surfaces such as beds, walls, hands, gloves, bandages, dressings, catheters, endoscopes, hospital equipment, sanitary devices, and the like.” And paragraph [0071] The sample may comprise one or more of the following: …a surface…laboratory equipment, a sanitary device, a sanitary equipment, ambient air, a biochemical assay chip, a microfluidic chip,..).
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to further expand the Benouis/ Radhakrishnan pipeline to include sampling of abiotic materials for the detection of pathogen and to further classify pathogen detection as a contamination, a faulty defect. It would have been predictable to add pathogen detection capability on a abiotic sample in the Benouis/ Radhakrishnan to broaden the use of the system allowing for numerous applications of the system beyond food. It creates a one stop system for pathogen detection on various types of matter using hyperspectral imaging and AI processing.
As per claim 66
Claim 66 is the method claim of system claim 23 and will be rejected under the same premise.
As per claim 67
Claim 67 is the method claim of system claim 24 and will be rejected under the same premise.
As per claim 84
Claim 84 requires a system with one or more processors and a non-transitory computer readable medium with instructions for a processor to implement the method of claim 41. Claim 41’s methods limitations were previously rejected by Benouis in claim 41’s 102 rejection. Benouis will not be relied upon for the actual hardware componentry of claim 84.
Radhakrishnan has one or more processors and a non-transitory computer readable medium connected to the one or more processors having instructions thereon which when executed by the one or more processors cause the one or more processors to perform a method (Paragraph [0075] “A person skilled in the art will readily recognize that steps of the method 400 can be performed by programmed computers. Herein, some examples are also intended to cover program storage devices and non-transitory computer readable medium, for example, digital data storage media, which are computer readable and encode computer-executable instructions, where said instructions perform some or all of the steps of the described method 400. The program storage devices may be, for example, digital memories, magnetic storage media, such as magnetic disks and magnetic tapes, hard drives, or optically readable digital data storage media.”
Accordingly, a person of ordinary skill in the art at the time this invention was effectively filed would have been motivated to implement Benouis’ hyperspectral defect detection approach within the computing architecture of Radhakrishnan. Benouis already teaches detecting defects by analyzing wavelength dependent spectral signatures in hyperspectral image cubes using ML classifiers (see claim 41’s 102 rejection)while Radhakrishnan teaches implementing spectral analysis algorithms using processors executing stored instructions within a computing system. Incorporating the Benouis hyperspectral defect classification pipeline into a processor-based computing architecture would have been a predictable implementation choice for executing the spectral analysis and classification algorithms required by Benouis.
Claim 33 and 76 are rejected under 35 U.S.C. 103 as being unpatentable over Benouis et al (Benouis et al hereinafter, Food Tray Sealing Fault Detection in Multi-Spectral Images Using Data Fusion and Deep Learning Techniques) in view of Ooi et al (Ooi hereinafter US 10823615 B2).
As per claim 33
Benouis teaches all claim limitations previously rejected in claim 1’s 102 rejection. See claim 1’s 102 rejection.
Benouis will not be relied upon for the hyperspectral imager having an array of unique wavelength filter lenses.
Ooi teaches the hyperspectral array imager comprising an array of unique wavelength filter lenses (Abstract: “the hyperspectral imaging apparatus includes a micro-lens array having a plurality of micro-lenses; and a filter array having a plurality of tunable filters…Each micro-lens and the corresponding coupled tunable filter are configured to generate a spectrally filtered image of a scene…Each of the plurality of tunable filters is tunable to transmit a selected wavelength within a respective spectral band, wherein the spectral bands of the plurality of tunable filters are different from each other.”)
A person of ordinary skill in the art at the time this invention was effectively filed would have found it obvious to substitute Benouis’ imager for Ooi’s hyperspectral imager which has unique wavelength filer lenses. This is because the same type of hyperspectral data required by Benouis' analysis pipeline is available with Ooi’s imager and has the advantage of providing a variety of snapshot individual hyperspectral images.
As per claim 76
Claim 76 is the method claim of system claim 33 and will be rejected under the same premise.
Claim 43 is rejected under 35 U.S.C. 103 as being unpatentable over Benouis et al (Benouis et al hereinafter, Food Tray Sealing Fault Detection in Multi-Spectral Images Using Data Fusion and Deep Learning Techniques) in view of Ni et al (Ni hereinafter CN 111753121 A).
Benouis teaches all claim limitations previously rejected in claim 41’s 102 rejection. See claim 1’s 102 rejection
Benouis teaches per pixel processing (Abstract: “Several pixel-based image fusion methods are proposed to obtain 2D images from the 3D hyperspectral image datacube, which feeds the deep learning (DL) algorithms.”) Benouis also discloses the hyperspectral cube’s structure in the introduction “For each image frame, data cubes generated by HSI sensors contain 3D data (𝑊×𝐻×𝐷) corresponding to an image in size of W pixels in width, H pixels in height, and D spectral channels in depth, i.e., two dimensions in spatial information and one dimension in spectral information, usually called datacube. “ , the two-dimensional aspect corresponding to pixels. This shows that the pixel level data is used for classification.
Benouis is not relied upon for the concept of sub pixel level material classification.
Ni teaches classification at a sub pixel material level( Ni states within the Content of Invention section that “The technical solution provided by the invention can be seen, on the one hand, using the spectrum segment vector on the dictionary vector set of projection, namely the linear representation of the dictionary vector (sparse representation of spectral segment vector), realizing the mixed pixel of sub-pixel decomposition, so as to realize accurate interpretation and target identification…”. Ni Further states within the Specific implementation examples section “Those skilled in the art will understand that …the existence of the mixed image element will increase the difficulty of classification identification,” and the solution within “…the embodiment of the invention, learning based on spectral vector dictionary, establishing a unified mathematical model of multi-hyperspectral remote sensing image sub-pixel interpretation, using efficient dictionary learning algorithm as means, realizing high precision interpretation of the multi-hyperspectral remote sensing image and identification of the fuzzy micro target.”. Ni explains that the hyperspectral pixels often contain mixed spectral signals and the method decomposes those signals to identify targets within the pixel. This explicitly shows Ni works on classification/mapping (identification) at the sub pixel level.
Accordingly, a person of ordinary skill in the art at the time this invention was filed would have been motivated to incorporate the sub pixel recognition approach of Ni within the hyperspectral defect detection pipeline of Benouis because Benouis already analyzes spectral signatures within hyperspectral data cubes to identify anomalies while Ni teaches that hyperspectral pixels may contain mixed spectral signals and can be decomposed to identify targets within the pixel i.e. sub pixel level identification. Applying sub pixel level decomposition techniques, a to the pixel level spectral analysis of Benouis would predictably improve defect classification by allowing detection and classification of materials or defects that occupy a fraction of a pixel.
Claim 77 is rejected under 35 U.S.C. 103 as being unpatentable over Benouis et al (Benouis et al hereinafter, Food Tray Sealing Fault Detection in Multi-Spectral Images Using Data Fusion and Deep Learning Techniques) in view of Qiu et al (Qiu hereinafter CN 112733659 A High-spectrum Image Classification Method Based On Self-paced Learning Double-flow Multi-scale Dense Connection Network).
Benouis teaches all claim limitations presented in claim 41’s 102 rejection. See claim 41's 102 rejection.
Benouis is not relied upon for employing a feature recalibration module for enhancing content of interest in the images of the object.
Qiu teaches a hyperspectral image classification model that includes an attention based feature recalibration mechanism (Contents of invention: “In the invention, the local space spectrum feature extraction branch comprises an image block and a multilayer Ghost residual network, wherein the input of the multi-layer Ghost residual network is the image block in the training sample set I1, outputting the local space spectrum characteristic f1; the multi-layer Ghost residual network is composed of a plurality of Ghost residual units; SE attention module (Hu J, Shen L, Sun G. Squeeze-and-the IEEE conference on the computer vision and pattern: 2018: 7132-7141), 1 * 1 convolutional layer and average pool layer.” An SE attention module is used to recalibrate features channel wise by focusing on the similarities and dependencies between channels; suppressing less informative features and enhancing important ones.)
Accordingly a person of ordinary skill in the art at the time this invention was effectively filed would have modified Benouis’ workflow to include Qiu’s SE attention based feature recalibration module because Benouis already showcases deep neural network classification of hyperspectral data while Qiu teaches that integrating SE attention into hyperspectral classification network improves feature extraction by recalibrating the images to emphasize what’s important information both spectral and aspatial. Incorporation of this module into Benouis’ architecture would have improved the detection and classification of defects by enhancing salient features corresponding to the defects in the hyperspectral images while suppressing less relevant information.
Claim 44 is rejected under 35 U.S.C. 103 as being unpatentable over Benouis et al (Benouis et al hereinafter, Food Tray Sealing Fault Detection in Multi-Spectral Images Using Data Fusion and Deep Learning Techniques) in view of Grassucci et al (A QUATERNION-VALUED VARIATIONAL AUTOENCODER).
Benouis covers all claim limitations previously rejected with claim 41’s 102 rejection. See claim 41’s 102 rejection.
Benouis teaches the underlying classification mapping pipeline on hyperspectral data but will not be relied upon for deep hypercomplex based reversible data reduction.
Grassucci teaches deep hypercomplex based reversible data reduction (3.2 Network architecture: “For the scope of the paper, here we use a rather simple architecture in order to prove the benefits of the variational inference in the quaternion domain. Thus, we consider an encoder network composed of quaternion convolutional layers”) in regards to reducing data (Conclusion: “Moreover, the QVAE involves the quaternion convolutional layers in both the encoder and the decoder networks, which lead to an impressive reduction of the overall number of network parameters.” Note that Variational Autoencoders themselves reduce dimensions/compress data and a QVAE does so on a hypercomplex quaternion algebra level. ) In regards to “reversibility” Grassucci states the in the Abstract that the “variational autoencoders (VAEs) have proved their ability in modeling a generative process by learning a latent representation of the input” and in the Conclusion section that “the proposed proper QVAE is able to learn latent representations in the quaternion domain by leveraging the augmented second-order statistics of the quaternion-valued input.” The latent representation is used in a VAE generative reconstruction framework so the reduced representation encapsulates the original input data rather than discarding it allowing for reversibility criteria. The encoder performs the data reduction and the decoder reconstructs original signal from reduced representation. Thus, the latent representation is a reversible reduced representation of the input data.).
Accordingly, a person of ordinary skill in the art would have been motivated to incorporate the quaternion domain latent representation processing of Grassuci into the hyperspectral classification pipeline of Benouis. Benouis shows deep learning model classification of defect related hyperspectral image data while Grassuci teaches a deep hypercomplex model that learns a latent representation of the input and does so in a way that preserves critical (and or as needed) informational content needed for reconstruction and or generative recovery. Using Grassuci’s quaternion latent representation processing as an added front end or intermediate stage in Benouis pipeline would have provided a deep hypercomplex data reduction before classification workflow. This would allow preservation of original content within the reduced features supporting a reversible data reduction criterion processing for classification in a task agnostic manner.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHANE WRENSFORD CODRINGTON whose telephone number is (571)272-8130. The examiner can normally be reached 8:00am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHANE WRENSFORD CODRINGTON/Examiner, Art Unit 2667
/TOM Y LU/Primary Examiner, Art Unit 2667