DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Notice to Applicants
2. This communication is in response to the application filled on 12/31/2024.
3. Claims 1-6 are pending.
4. Limitations appearing inside {} are intended to indicate the limitations not taught by said prior art(s)/combinations.
Information Disclosure Statement
5. The information disclosure statement (IDS) submitted on 12/31/2024 has been considered by the examiner.
Claim Objections
6. Claim 5 is objected to because of the following informalities:
Claim 5 recites “…one or more of …(CNN) and a … (RNN).”. See MPEP 2111.01(II), citing Superguide Corp. v. DirecTV Enterprises, Inc. 358 F. 3d 870, 875, 69 USPQ2d 1865, 1868 (Fed. Cir. 2004).
Appropriate correction is required.
Double Patenting
7. The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claim 1 is provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claim 4 of copending Application No. 18/880,323 (hereinafter 323) in view of the prior art as applied below.
Specifically, the examiner notes claim 1 of 323 contains analogous recitations of hyperspectral image acquisition using a hyperspectral sensor, and dependent claim 4 of 323 contains analogous recitations a semi-supervised learning processing model and target object material discrimination unit. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to combine the below prior art to teach a hyperspectral data acquisition unit to obtain the invention as specified in claim 1 of the pending application. The examiner specifically notes that prior art teaches a “hyperspectral data acquisition unit for acquiring hyperspectral data on a target object by determining an analysis region from a hyperspectral image of waste”, and that the motivation for an “analysis region” determination would have been obvious in view of the hyperspectral characteristics of an object being the differentiating factor to determining if an object is specifically recognized as a target object (see citations below, specifically Sudharshan [pg. 11490, Abstract, par. 1, ln. 1-14]), and as such can allow for more effective discrimination.
This is a provisional nonstatutory double patenting rejection.
Claim Interpretation
8. The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitations uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are: “…a hyperspectral data acquisition unit…”, “…a semi-supervised learning processing model unit…”, and “…a target object material discrimination unit…”, in claim 1.
Because these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have these limitations interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitations to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitations recite sufficient structure to perform the claimed function so as to avoid them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
9. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
10. Claims 1, 3-5 are rejected under 35 U.S.C. 103 as being unpatentable over “Object Detection Routine for Material Streams Combining RGB and Hyperspectral Reflectance Data Based on Guided Object Localization” to Sudharshan et al. (hereinafter Sudharshan) and further in view of “Semi-supervised Deep Learning Techniques for Spectrum Reconstruction” to Simonetto et al. (hereinafter Simonetto).
11. Regarding Claim 1, Sudharshan teaches a hyperspectral image-based waste material discrimination system ([pg. 11490, Abstract, par. 1, ln. 1-14] “Electronic waste is the fastest growing type of scrap globally and is an important challenge due to its heterogeneity, intrinsic toxicity and potential environmental impact. With an objective of obtaining information on the composition of printed circuit boards (PCBs) through non-invasive analysis to aid in recycling and recovery of precious waste, the goal of this paper is to propose a scheme towards the fusion of RGB and hyperspectral data in object detection. State-of-art detectors come with their own set of challenges which make them inapplicable to PCB recycling. We introduce a method which promises to achieve object detection based on multi-sensor data by utilizing the hyperspectral data to localize components and compare the results to a conventional single-sensor (RGB) based approach.”, [pg. 11492, col. 1, Fig. 1 and 2, Table 1, see Specim FX17 Hyperspectral camera]), comprising:
a hyperspectral data acquisition unit for acquiring hyperspectral data on a target object by determining an analysis region from a hyperspectral image of waste, acquired through a hyperspectral sensor ([pg. 11492, col. 1, Fig. 1 and 2, Table 1, see Specim FX17 Hyperspectral camera], [pg. 11941, col. 2. A. Data Acquisition and Pre-Processing, par. 1, ln. 1 to pg. 11492, col. 2, par. 1, ln. 3] “The FX17 (Spectral Imaging Ltd) and the Teledyne Dalsa C4020 cameras which are mounted in a custom setup above a conveyor belt, were used to acquire the data from the material stream (Fig. 2). FX17 is a line scan hyperspectral camera with high data acquisition rate and relatively high spatial resolution covering a relevant range within SWIR chosen for its high signal-to-noise ratio while the C4020 is a full frame-capture RGB camera. The camera parameters used for the data acquisition are given in table I. The sample was illuminated by four broad-band quartz-tungsten halogen units without protective cover. The raw data was acquired using the GigE interface, a standardized data transfer technology. For HSI, the dark frames are captured with the shutter closed prior to the sample data. The data acquisition of each sample is preceded by a white reference panel (99% reflectance), a gray reference panel (50%reflectance) and a black reference panel (6% reflectance). The hyperspectral camera captures the data in one spatial and one spectral dimension. In order to obtain the hypercube from the raw data, the software routine ProMISE [16] devel oped at HIF was utilized. The background noise from the HS cameras is subtracted from the raw data. Following this, the program uses the white, gray and dark reference levels to calculate the radiance-to-reflectance conversion and stacks the data to form a hypercube with the pixels containing the reflectance spectra. The resulting hypercube can be annotated for pixelwise classification. The hypercube is stored in the form of a hdr file. As the RGB camera utilizes a Bayer pattern on the sensor to capture the data, the raw image was first debayered and then converted to LAB color-space. In order to enhance the definition of edges and to improve the contrast of the image, Contrast Limited Histogram Adaptive Equalization (CLAHE) [17] was performed using the OpenCV module of python. As we are working with fusion of RGB and HSI data, co-registration [18] is performed on the two modalities which ensures that the images become spatially aligned so that any feature in one modality overlaps its footprint in the corresponding image of a different modality. The preprocessed image can be annotated for ground truth data.”);
a {semi-}supervised learning processing model unit for generating integrated data by processing the hyperspectral data through a {semi-}supervised learning processing model ([pg. 11491, col. 2. A. Data Acquisition and Pre-Processing, par. 1, ln. 1 to pg. 11492, col. 2, par. 1, ln. 3], [pg. 11492, col. 2, B. Pixelwise Classification Using EMP, par. 1, ln. 1 to pg. 11493, col. 1, par. 1, ln. 3] “We perform pixelwise classification in order to obtain classification maps for later integration with RGB pre-processed data. We use an SVM with an RBF (radial-basis function) kernel (for the sake of simplicity we call this method kernel-SVM from now on). It outperforms classic supervised learning algorithms [19]–[21] such as maximum-likelihood and k-nearest neighbors and its ability to deal with a limited number of training samples and a high number of spectral bands, both of which apply to our dataset. As kernel-SVM is a supervised learning approach, we require annotated/labeled data for the purposes of training. Using the Spectral Python (SPy) module, a code was developed for pixelwise annotation of data. As kernel-SVM is not scale invariant, the first step of training a classifier is to pre-process the data by feature scaling. Feature scaling ensures that the other features in the model are not dominated by the features with greater orders of magnitude. Since we deal with 3-dimensional data i.e. two spatial dimensions and a spectral dimension, the HSI is flattened before using a standard scaler algorithm. The features are scaled to a range centered around zero so that the feature variances are in the same range. Following the feature scaling, dimensionality reduction is applied to the data using principal component analysis, a widely known approach chosen for its low execution time [22]. The first five principal components are considered as they provide a good balance between speed and accuracy in the form of accumulated variance. A number of methods have been proposed to integrate spectral and spatial features into a classification framework to boost the quality of the classification map. Among those approaches, the ones that are based on mathematical morphology (differential morphological profiles (DMP) [23], extended morphological profiles (EMP) [24], and weighted DMP [25] have gained popularity due to their simplicity in terms of implementation and processing time as well as their effectiveness in terms of classification performance. EMP applies a feature extraction method (such as PCA) to reduce the dimensions of the HSI and redundancy between the bands. EMPs are derived for each of the principal components. Finally, the dataset is split in training and testing data which is used to train the kernel-SVM classifier to obtain the classification map shown in Fig. 4.”); and
a target object material discrimination unit for discriminating the material of the target object through a deep learning model on the basis of the integrated data ([pg. 11493, col. 1, C. Faster-RCNN, par. 1, ln. 1 to pg. 11494, col. 1, par. 3, ln. 9] “As mentioned earlier, the goal of this network is to localize components on a PCB. For this purpose, the object-detector Faster-RCNN (Regional-CNN) was used. This is because Faster-RCNN [26] is a two-stage detector which uses a region proposal network to generate anchors, a feature which is utilized in our model to introduce HS data. The simplicity of the network is an advantage in this aspect. Faster RCNN also performs better than many other object detectors in terms of detecting small objects [27] which would be useful in our use case scenario to detect PCB components. The goal of Faster-RCNN is threefold. Given an image, the network is required to output a list of bounding boxes, a label assigned to a bounding box and the corresponding probabilities for the label. In order to accomplish this, Faster RCNN utilizes four sub-components. These are the CNN for feature map extraction, RPN for region proposal generation, region of interest (ROI) pooling for proposal selection, and a classifier for prediction. The aforementioned sub-components are visualized in Fig. 3. The first step in Faster-RCNN is the extraction of feature maps using CNN. The output of an intermediate layer pretrained for classification is used for this purpose. We use VGG-16 [28] as the feature extraction network… The extracted feature map is then used by the RPN to generate object proposals (explained in the following section). The extracted object proposals need to be classified. Faster-RCNN reuses the existing convolutional feature map and extracts fixed size feature maps for each proposal by cropping the feature map. This stage is called ROI pooling. These extracted feature maps are followed by fully-connected layers in the classification layers, mimicking a CNN based classifier. The output is the final set of refined bounding box proposals and the classes along with the confidence score, i.e. how confident the network is in its prediction… This section explains the object detection algorithm using HSI-guided object localization. The dataset used consists of the preprocessed and annotated RGB data and the classification maps which are the predictions of the trained pixel wise classification network. Morphological operators are then applied on the classification map to remove noise from the predictions. The getData function is altered to parse the classification map which is a two dimensional array. During data augmentation, the classification maps are transformed to conform to the augmented data along with the bounding boxes… The region proposal network (RPN) employs a single con volution layer with sigmoid activation on the feature map following which the objectness scores are calculated for each anchor in the feature map. The pixelwise classification map predicts a score map
p
(
.
|
F
I
)
which has the same size as the feature map
F
I
. Each entry of this binary map
p
(
i
,
j
|
F
I
)
can be translated onto the input image to the coordinates
(
i
+
1
2
s
,
j
+
1
2
s
)
where s is the stride of the CNN. This value predicts the objectness score at that location…”).
Sudharshan does not specifically disclose wherein the learning processing model for generating integrated data is a semi-supervised model. Specifically, Sudharshan uses a supervised model in an SVM.
However, Simonetto teaches wherein the learning processing model for generating integrated data is a semi-supervised model ([pg. 7768, col. 1, par. 1, ln. 1 to col. 2, par. 2, ln. 18] “As already pointed out, the biggest issue when trying to exploit machine learning for this task is the lack of hyperspectral data. As a matter of fact, while there exist databases of labeled RGB images for various tasks containing tens of thousand or even millions of samples, the biggest hyperspectral images database is composed of only 256 images [14]. Training a Deep Neural Network with such a limited amount of data is a very challenging task easily leading to overfitting on the training set. A commonly used workaround is to split the training images into patches: this strategy allows to reduce the issue but goes far from solving the problem. Moreover, these large datasets are only available for a few specific cases and it is therefore impossible to deploy such networks in other situations without creating a large HSI dataset which is costly and time-consuming. In this paper we introduce a set of learning strategies able to provide good performances even when the HSI data at our disposal is very limited, thus overcoming the issue. We consider 4 possible scenarios… The second (Ti) is a semi-supervised scenario where a large number of RGB images are at our disposal but only a few hyperspectral ones. In the third setting (Tp) we have the RGB images while the HSI information is given only for a few pixels. Notice that this is a much more challenging setting compared to the previous one since the number of samples with hyperspectral information is several orders of magnitude smaller, but it is interesting since it matches the behaviour of point spectrometers. For the latter we consider a domain transfer setting where HSI information is available for a source domain but only a limited amount of hyperspectral samples are provided for the target domain, that has slightly different properties”, [pg. 7769, col. 2, C. Semi-supervised Training with Images (Ti), par. 1, ln. 1 to pg. 7770, col. 1, par. 2, ln. 16] “When the training dataset contains both RGB and hyperspectral images, the previous approach can be combined with the standard supervised training on hyperspectral data. The training in this case is performed in two phases. In the first only RGB images are used and an initial model is trained for
S
1
steps in an unsupervised way as in Section IV-B, minimizing the loss
L
u
. This initial stage performs a robust initialization of the network making the semi-supervised training in the next phase more stable. In the second phase we use together the two different sets of images, the RGB ones and the hyperspectral ones. For images where only RGB data is available, we use the previously introduced physical model loss, but without the total variation term. We disabled this term since while in the initial phase its smoothing effect was needed for a more stable estimate, in the second phase it would lead to an excessive smoothing of the spectra, as a more accurate shape can now be learned from the HSI data. For images where the HSI data is available, we instead use a supervised strategy where we compare the HSI ground truth with the prediction using the Mean Relative Absolute Error (MRAE):
L
H
S
I
=
1
N
i
N
j
N
s
∑
i
=
1
N
i
∑
j
=
1
N
j
∑
s
=
1
N
s
|
P
i
,
j
,
s
-
H
i
,
j
,
s
|
H
i
,
j
,
s
where
N
s
is the number of spectral bands (31 in our case), H denotes the ground truth hyperspectral values and P the network prediction. We used the MRAE since this metric normalizes the error w.r.t. the intensity of the ground truth, avoiding a bias towards brighter pixels (as happening with RMSE), especially because it is common for some of the channels to be consistently darker than others. In particular, each batch of training data (that contains Bs = 64 samples in our experimental setting, see Section VI-A) contains only images from one set (i.e., with HSI ground truth or RGB only). Assuming to have a much larger amount of RGB images than hyperspectral data, the training procedure is organized in blocks of k iterations: each block consist in a first iteration where the network is trained on a batch of
B
s
patches with HSI ground truth followed by k −1 iterations where the network is trained on RGB only data (the RGB batches are different in each iteration). The training procedure is shown in Fig. 3. It is repeated for S2 steps to obtain the final model. Notice how the use of RGB images in training reduces the risk of overfitting: if the network overfits on the HSI images, which are few, still it will have to fulfil the physical model constraint on the much larger RGB dataset.”, [pg. 7770, Fig. 3]). Specifically, one of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Sudharshan and Simonetto as within the same field of image processing and classification using duel RGB hyperspectral machine learning approaches, and as analogous to the claimed invention. The motivation to combine is disclosed in Simonetto, in that using a semi-supervised approach as opposed to supervised approach prevents overfitting and prevents the need to create large hyperspectral datasets ([pg. 7768, col. 1, par. 1, ln. 1 to col. 2, par. 2, ln. 18]). This specifically addresses one of the issues noted in Sudharshan, which is a lack of training data with regard to recycling waste ([pg. 11491, col. 2, par. 2, ln. 1-10] “The here presented study addresses the need for combining the advantages of RGB and HSI data and the challenges imposed by limited training data in the use case scenario of PCB object detection for recycling applications. As mentioned earlier, the significantly low amount of training data results from the early stage of research on applications of multi-sensor systems to PCB recycling. All the training and test data required for the model proposed in our study were acquired using the multi-sensor system at HIF. The limited availability of PCBs results in a small training dataset. Thus, conventional multi-modal neural networks cannot be used in this scenario.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the system of Sudharshan with the semi-supervised training approach of Simonetto through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, one of ordinary skill in the art would have further incorporated an analogous semi-supervised training of a learning processing model as taught in Simonetto for the pixelwise classification of Sudharshan so as to further compensate for the lack of training data and remove the requirement of creating a hyperspectral dataset to train with.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the system of Sudharshan with the semi-supervised learning processing model of Simonetto to obtain the invention as specified in claim 1.
12. Regarding Claim 3, a combination of Sudharshan and Simonetto teaches the system of claim 1. Rejections analogous to claim 1 are further applicable to claim 3. Specifically, Sudharshan teaches wherein the hyperspectral data comprises labeled data {and unlabeled data}, and wherein the {semi-}supervised learning processing model processes the labeled data {and the unlabeled data} through a principal component analysis network ([pg. 11492, col. 2, B. Pixelwise Classification Using EMP, par. 1, ln. 1 to pg. 11493, col. 1, par. 1, ln. 3]). Sudharshan does not specifically disclose semi-supervised learning or unlabeled data.
However, Simonetto teaches wherein the learning processing model for generating integrated data is a semi-supervised model and unlabeled data ([pg. 7768, col. 1, par. 1, ln. 1 to col. 2, par. 2, ln. 18], [pg. 7769, col. 2, C. Semi-supervised Training with Images (Ti), par. 1, ln. 1 to pg. 7770, col. 1, par. 2, ln. 16] [pg. 7770, Fig. 3]). The motivation to combine remains analogous to claim 1 ([pg. 7768, col. 1, par. 1, ln. 1 to col. 2, par. 2, ln. 18]). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the system of Sudharshan with the semi-supervised training approach using unlabeled data of Simonetto through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the system of Sudharshan with the semi-supervised learning processing model and unlabeled data of Simonetto to obtain the invention as specified in claim 3.
13. Regarding Claim 4, a combination of Sudharshan and Simonetto teaches the system of claim 3. Sudharshan teaches wherein the hyperspectral data comprises spatial information and spectral information ([pg. 11941, col. 2. A. Data Acquisition and Pre-Processing, par. 1, ln. 1 to pg. 11492, col. 2, par. 1, ln. 3] see hypercube and coregistration to spatially align the modalities); and wherein the {semi-}supervised learning processing model unit generates the integrated data by integrating respective results obtained after training each of the spatial information and the spectral information through the {semi-}supervised learning processing model ([Fig. 4] see spectral scores corresponding to pixel locations, [pg. 11492, col. 2, B. Pixelwise Classification Using EMP, par. 1, ln. 1 to pg. 11493, col. 1, par. 1, ln. 3]). Sudharshan does not specifically disclose wherein the model is semi-supervised.
However, Simonetto specifically discloses wherein the learning processing model for generating integrated data is a semi-supervised model ([pg. 7768, col. 1, par. 1, ln. 1 to col. 2, par. 2, ln. 18], [pg. 7769, col. 2, C. Semi-supervised Training with Images (Ti), par. 1, ln. 1 to pg. 7770, col. 1, par. 2, ln. 16] [pg. 7770, Fig. 3]). Specifically, the examiner notes that Simonetto uses an analogous methodology that includes both spatial and spectral information ([pg. 7769, col. 1, A. Physical model, par. 1, ln. 1-31] “In the case of spectrum reconstruction, the underlying physical model that can be exploited is the conversion of hyperspectral images back to the RGB domain (see Fig. 1): while the RGB to hyperspectral mapping is very under constrained, the opposite relation is an univocal function
f
H
S
I
→
R
G
B
… For this reason we mapped the output of the network back to the RGB space and compared it to the input image. The difference between the two images is captured by the loss
L
R
G
B
:
L
R
G
B
=
1
3
N
j
N
s
∑
i
=
1
N
i
∑
j
=
1
N
j
∑
c
=
1
3
|
I
i
,
j
,
c
-
f
H
S
I
→
R
G
B
c
-
P
i
,
j
|
where we denote the pixels in the original image with I and the predicted HSI samples with P. Ni and Nj correspond to the size of the image in the spatial dimensions and c loops over the 3 color channels. In a practical case the conversion would need to be done using the camera spectral sensitivity functions, which are usually provided or known. In our case, since there are no aligned RGB images in the datasets, we employed a standard conversion from the HSI domain to the sRGB domain. We used the CIE 1931 colour matching functions, assumed a D65 standard illuminant and applied gamma correction. However notice that
f
H
S
I
→
R
G
B
can be any generic function and the proposed approach is agnostic to the employed camera model. As discussed the physical model loss alone is far from sufficient for reliable results and additional constraints are needed. In fact, there is a multitude of spectra which can be converted back to the same RGB values, as the transformation is the output of an integration operation. The physical model does not help in resolving this ambiguity, but penalizes all options not leading to the input RGB values.”). The motivation to combine remains analogous to claim 1. One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the system of Sudharshan with the semi-supervised training approach using unlabeled data of Simonetto through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the system of Sudharshan with the semi-supervised learning processing model and unlabeled data of Simonetto to obtain the invention as specified in claim 4.
14. Regarding Claim 5, a combination of Sudharshan and Simonetto teaches the system of claim 3. Sudharshan further teaches wherein the deep learning model uses one or more of a convolutional neural network (CNN) and a recurrent neural network (RNN) ([pg. 11493, col. 1, C. Faster-RCNN, par. 1, ln. 1 to pg. 11494, col. 1, par. 3, ln. 9]). Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the system of Sudharshan with the semi-supervised learning processing model and unlabeled data of Simonetto to obtain the invention as specified in claim 5.
15. Claim 2 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over “Object Detection Routine for Material Streams Combining RGB and Hyperspectral Reflectance Data Based on Guided Object Localization” to Sudharshan, view of “Semi-supervised Deep Learning Techniques for Spectrum Reconstruction” to Simonetto, and further in view of U.S. Publication No. 2022/0331841 to Filler et al. (hereinafter Filler).
16. Regarding Claim 2, a combination of Sudharshan and Simonetto teaches the system of claim 1. Sudharshan further discloses wherein the hyperspectral data acquisition unit specifies the target object by considering locations of a vision camera and the hyperspectral sensor ([pg. 11941, col. 2. A. Data Acquisition and Pre-Processing, par. 1, ln. 1 to pg. 11492, col. 2, par. 1, ln. 3]), {and a moving speed and a moving distance} of the waste on a conveyor belt ([pg. 11941, col. 2. A. Data Acquisition and Pre-Processing, par. 1, ln. 1 to pg. 11492, col. 2, par. 1, ln. 3]), and determine the analysis region of the target object ([pg. 11491, col. 2. A. Data Acquisition and Pre-Processing, par. 1, ln. 1 to pg. 11492, col. 2, par. 1, ln. 3], [pg. 11492, col. 2, B. Pixelwise Classification Using EMP, par. 1, ln. 1 to pg. 11493, col. 1, par. 1, ln. 3]) {by excluding a portion in which the target object overlaps with other waste}.
Sudharshan and Simonetto do not specifically teach wherein the target object is specified considering a moving speed and moving distance of the waste on a conveyor belt, or wherein the analysis region is determined by excluding a portion in which the target object overlaps with other waste.
However, Filler specifically teaches wherein the moving spend and moving distance of the waste conveyor belt is considered for specifying a target object ([Fig. 1], [par. 0124, ln. 11-18] “Another is similar, but only checks incoming camera data against the start of the previous map data. Once a match with the start of the map is found, tracking begins. In all such cases the speed of the belt can be sensed, e.g., by determining the advance of the image data, in pixel rows over a series of frames captured at a known rate (e.g., 300 fps). Keypoint detection can be employed, to identify corresponding points in belt images separated by one or more frame intervals.”, [par. 0166, ln. 1-11] “The map data generated by the AI system and communicated to the watermark system can be specified in terms of pixel locations within the AI system camera field of view. Alternatively, such pixel locations can be mapped to corresponding physical coordinates on the conveyor belt (such as at a position 46.5 feet from a start-of-belt marker, and 3 inches left of belt center line.) Given a known belt speed and a known distance between the AI and watermark system cameras, the mapping to corresponding pixel locations within the watermark system camera field of view is straightforward.”), and wherein the analysis region is selected by excluding a portion in which the target object overlaps with other waste ([par. 0322, ln. 1-16] “Although waste items are usually distributed across a conveyor belt in isolated (singulated) fashion, with empty areas of belt separating items, this is not always the case. When two waste items touch (adjoin) or overlap, they can be mistaken for a single item. A determination of attribute information (e.g., plastic type, or food/non-food, etc.) about a first item at one point on the conveyor belt (e.g., as when a patch of watermark signal or a NIR signature at one location indicates a particular type of plastic) can thus be mis-attributed to waste occupying an adjoining region of belt that is actually a second item. Both items may be ejected together into a collection bin, impairing purity of the items collected in that bin. Or, attempted air jet diversion targeted to a central point within the collective area occupied by the two items can deflect the two items in unexpected directions, again leading to undesired results.”, [par. 0323, ln. 1-26] “As referenced earlier, a region growing algorithm can be employed to determine the physical area on a belt occupied by an item. Region growing algorithms are familiar to image processing artisans. Other names for such processes are blob extraction, connected-component labeling, and connected component analysis. An exemplary region growing algorithm starts with a seed pixel, which is assigned a label (e.g., an object ID, such as an integer number). Each pixel that adjoins the seed pixel is examined to determine if it has a particular attribute in common with the neighboring seed pixel. In the present case, this attribute can be a sensed NIR response indicative of non-belt. In one example, if the neighboring pixel has an 8-bit greyscale value below 15 in each of the sensed NIR wavelengths, it is regarded as depicting the conveyor belt; else such value indicates non-belt (i.e., waste on the belt). Those neighboring pixels that are indicated as non-belt are assigned the same label as the original seed pixel. This process continues from each of the just-examined pixels that were labeled in common with the original seed pixel. In this fashion, regions of imagery contiguous to pixels having a particular labeled attribute are progressively-explored and labeled in common with the seed pixel until an outer boundary is reached where no other pixel adjoining labeled pixels meets the tested attribute. The resulting collection of labeled pixels defines a contiguous area apparently spanned by an object on the belt.”). One of ordinary skill in the art, before the effective filling date of the claimed invention, would recognize Sudharshan, Simonetto, and Filler as within the same field of image processing and classification using duel RGB hyperspectral machine learning approaches, and Sudharshan and Filler as further within the same field of hyperspectral classification for waste management systems, and as analogous to the claimed invention. The motivation to combine is disclosed in Filler, wherein the speed and distance of the belt can be used as a straightforward and simple means to determine the target object between the two displaced sensors ([par. 0166, ln. 1-11]), and wherein excluding a portion of overlapping objects prevents undesirable objects from being selected for recycling and/or ejection ([par. 0322, ln. 1-16]). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the system of Sudharshan with the semi-supervised training approach of Simonetto, and further combined the system of the combination of Sudharshan and Simonetto with the belt speed and distance and exclusion of overlapping portions of the target object as taught in Filler through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, one of ordinary skill in the art would have combined the system of the combination of Sudharshan and Simonetto with Filler such that it used the belt speed and distance as taught in Filler to obtain an analogous spatially aligned image for both modalities, and the exclusion of overlapping portions of the target object as taught in Filler to prevent erroneous object selection for recycling.
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the system of Sudharshan with the semi-supervised training approach Simonetto and the belt speed and distance and exclusion of overlapping portions of the target object of Filler to obtain the invention as specified in claim 2.
17. Regarding Claim 6, a combination of Sudharshan and Simonetto teaches the system of claim 1. While Sudharshan teaches that HSI typically includes shortwave infrared (SWIR) wavelengths ([pg. 11491, col. 1, par. 1, ln. 2-8] “As opposed to RGB images which capture information in three discrete channels, hyperspectral sensors have the ability to acquire image data in several hundreds of consecutive spectral bands over the electromagnetic spectrum, typically covering the visible and short-wave infrared (0.38μm-2.5μm) range of the electromagnetic spectrum.”), Sudharshan does not specifically disclose wherein the hyperspectral sensor specifically uses near infrared (NIR) or shortwave infrared (SWIR). Likewise, Simonetto does not specifically teach the hyperspectral sensor specifically uses near infrared (NIR) or shortwave infrared (SWIR).
However, Filler specifically teaches wherein the hyperspectral sensor uses NIR ([par. 0322, ln. 1-16], [par. 0323, ln. 1-26]). The motivation to combine would have been obvious to one of ordinary skill in the art, in that NIR allows for specific detections of attribute information for objects that allow for classification and differentiation form other overlapping objects ([par. 0323, ln. 1-26]). One of ordinary skill in the art, before the effective filling date of the claimed invention, would have combined the system of Sudharshan with the semi-supervised training approach of Simonetto, and further combined the system of the combination of Sudharshan and Simonetto with the NIR wavelength as taught in Filler through known means, with no change to their respective function, and the combination would have yielded nothing more than predicable results. Specifically, one of ordinary skill in the art would combined the system of the combination of Sudharshan and Simonetto and the NIR Filler to detect plastics (e.g., plastic on the PCB boards of Sudharshan).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filling date of the claimed invention, to combine the system of Sudharshan with the semi-supervised training approach Simonetto and the NIR wavelength of Filler to obtain the invention as specified in claim 6.
Conclusion
18. The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. See PTO-892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAULO ANDRES GARCIA whose telephone number is (703)756-5493. The examiner can normally be reached Mon-Fri, 8-4:30PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached on (571)272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PAULO ANDRES GARCIA/Examiner, Art Unit 2669 /CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights