DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments see Pages 5-12, filed 06/16/2025, with respect to claims 1 and 3 under 35 USC 103 have been fully considered but they are not persuasive.
Applicant argues on Pages 5-12 that Schwartzer, Izquierdo, Rai and Hollenbeck, either alone or in combination, fail to teach or suggest some, or all the recited features in independent claim 1, with emphasis on the following three features:
Feature 1: thermal imaging device is provided with a motor configured to rotate 360 degrees to assist said thermal imaging device to scan and capture the thermal images of said one or more fruits from all directions;
Feature 2: classifying the received thermal images of the one or more fruits into a plurality of classes according to a number of days the fruit will remain edible as particular fruit's shelf-life, wherein classifying the received thermal images is done by: comparing detailed temperature distribution with pre-defined threshold values also called as weights obtained by a trained model, by applying transfer learning, which is a deep learning technique, on a thermal dataset of a same standard fruit;
Feature 3: using the transfer learning, the weights of the pre-trained model are updated and are then used to classify a new thermal image data set which includes a thermal image of any new fruit to be tested for edibility in terms of number of days, and wherein features of the thermal image of the new fruit are extracted from convolution and pooling layers of the pre-trained model.
Specifically, in reference to Feature 1, Applicant argues that Schwartzer teaches “a handheld imaging device without any motorized mechanism for 360-degree rotation to capture images from all directions” and that Hollenbeck, introduced to overcome the deficiencies of Schwartzer in view of Izquierdo and Rai, teaches a 3D scanner, with “at best two cameras”, which provides a motorized means for varying the degrees of freedom of a platform on which an object sits relative to fixed cameras, but does not teach a thermal imaging device provided with a motor configured to rotate 360 degrees.
In response to Applicant’s arguments, the Examiner notes that Schwartzer’s handheld device provides a manual means of capturing images from all directions (Schwartzer: Fig. 5) and that Rai, in the same field of endeavor of systems and methods of monitoring perishable food items, which was introduced to modify Schwartzer in view Izquierdo with a scanning system and method using machine learning, discloses a system of fixed cameras within an custom enclosure which are able to capture images from all directions (Rai: [0034], lines 12-17). In view of the imaging systems noted in the cited prior art, it would have been obvious to one of ordinary skill in the art that capturing images of fruit objects for classifying the edibility of the fruit object would require complete scans to ensure quality. Though the examiner agrees that the mechanical apparatus used to capture complete scans, e.g., the motor that rotates a camera 360 degrees, is not disclosed by the cited prior art, it would be recognized that any means used to automate what may otherwise be a laborious manual process is not in itself an inventive concept. Therefore, the Examiner maintains that the introduction of the motorized platform of Hollenbeck, which can move the fruit object relative to fixed cameras would be an equivalent alternative to the claimed rotating 360-degree camera, and would yield similar results.
In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Specifically, Applicant argues that Izquierdo fails to disclose Feature 2 outlined above, see Page 8-10, and only discloses “use of convolutional neural networks for analyzing thermographic images” but is limited to detecting adulterations in food products. However, Izquierdo was merely introduced to show that use of CNNs in the analysis of food products is known in the art, and that Schwartzer may be modified with such a feature, providing a machine learning model which is optimized for such image processing applications. Regarding arguments against the motivation and that it would not be obvious to modify Schwartzer in view of Izquierdo, Applicant has only presented an argument that Izquierdo teaches a different application of thermal imaging and CNNs to detecting adulteration in food products, which is addressed by the above, and has not presented reasoning against the motivation provided in the Office Action that introducing machine learning allows for improved inference and generalization capabilities and increased accuracy of the measurement device. Further, the Examiner maintains that such a motivation would be known and obvious to one of ordinary skill in the art.
As for Feature 3, Applicant acknowledges that Rai discloses the use of transfer learning in food monitoring but that Rai does not teach “applying transfer learning specifically to thermal images of fruits to predict shelf life” and that Rai is silent on the aspect of classifying new thermal image data sets for edibility in terms of number of days. However, it should be noted that the Rai reference should be considered in combination with Schwartzer and Izquierdo, which combined teach Feature 3. Further, in view of the combination presented, it would also be obvious to one of ordinary skill in the art that determining shelf life quantitatively would imply testing and classifying fruit objects for edibility in terms of number of days (see Fig. 7A-B, 10A-B, etc. which show that fruit ripeness is being measured against number of days). Further, regarding Applicant’s argument on page 12 that “the combination does not disclose, teach, or suggest the feature using the transfer learning, the weights of the pre-trained model are updated and are then used to classify a new thermal image data set which includes a thermal image of any new fruit to be tested for edibility in terms of number of days, and wherein features of the thermal image of the new fruit are extracted from convolution and pooling layers of the pre-trained model”, the Examiner respectfully disagrees. The Applicant’s claim recites features that are inherent to classification algorithms using CNNs including updating weights, classifying new image data based on a trained model, and extracting features of new image data from convolutional and pooling layers. The implementation of the specific modifications transfer learning introduces would be known and obvious to make to those in the art. Therefore, for the reasons mentioned above, the rejections of claim 1 and 3 are maintained.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 3 are rejected under 35 U.S.C. 103 as being unpatentable over Schwartzer et al. (US 2019/0340749) in view of Izquierdo et al. (ES 2741073 A1, portions of the attached machine translations are cited) in view of Rai et al. (US 2020/0251229 A1) further in view of Hollenbeck et al. (US 2021/0005017 A1).
Regarding claims 1 and 3,
For claim 1: Schwartzer discloses a system for determining shelf life of a fruit, said system comprising (Fig. 1A, 1B, 7 and 8A; [0144], lines 1-4; [0170], lines 1-6):
a receiver (135) configured to receive an input from a user (Fig. 1A, 1B, 7 and 8A; [0112], lines 1-5) to take images of one or more fruits whose shelf life is to be ascertained ([0109], lines 4-10; [0141], lines 4-7; [0142]);
a thermal imaging device (166) configured to capture thermal images of one or more fruits on receiving the input by the first receiver (135), wherein said thermal imaging device captures thermal images of said one or more fruits (Fig. 1A, 1B, 7 and 8A; [0144] , for the purpose of this examination, the thermal imaging device is being interpreted by the examiner as a device capable of sensing infrared radiation and thermal images are being interpreted as images produced by capturing infrared radiation that may be either reflected off of an object or radiated by the object);
a transmitter (160) operationally interconnected to the thermal imaging device (166) to transmit the captured images by said thermal imaging device to a processing unit (150) (Fig. 1A, 1B, 7 and 8A; [0115], lines 1-5; [0151], lines 3-20 – where the examiner is interpreting the network as inherently comprising any number of first transmitters for the transfer of data from the imaging device to the processing unit);
the processing unit (150) (Fig. 1A, 1B, 7 and 8A; [0106], lines 4-9) configured to ascertain shelf life of said one or more fruits based on the received thermal images, wherein said processing unit is configured to ascertains shelf life of the one or more fruits ([0173], lines 1-5; [0151], lines 3-20) by:
classifying the received thermal images of the one or more fruits into a plurality of classes according to a number of days the fruit will remain edible as particular fruit's shelf-life ([0173], lines 1-5), wherein classifying the received thermal images is done by: image processing and machine learning ([0140], lines 7-13; [0165], lines 1-4 – where the suggested use of machine learning algorithms implies an automated process of classifying fruits);
a display unit to display determined shelf life of the fruit ([0112], lines 5-7; [0141], lines 4-7; [0142]).
Schwartzer does not disclose
that the thermal imaging device captures images from all directions,
a motor configured to rotate 360 degrees to assist said thermal imaging device to scan and capture the thermal images of said one or more fruits from all directions, wherein said thermal imaging device is configured to converts infrared radiation released by the fruit and surrounding of the fruit into a visible thermal image;
that the processing unit is configured to ascertains shelf life of the one or more fruits by:
determining the maximum and minimum temperature value of the fruit along with the temperature distribution of an entire fruit to differentiate whether scanned fruit is from cold storage or normal temperature automatically; and
classifying the received thermal images of the one or more fruits into a plurality of classes according to a number of days the fruit will remain edible as particular fruit's shelf-life, wherein classifying the received thermal images is done by:
comparing detailed temperature distribution with pre-defined threshold values also called as weights obtained by a trained model, by applying transfer learning, which is a deep learning technique, on a thermal dataset of a same standard fruit on the processing unit to determine the shelf life of the fruit, wherein the thermal dataset comprises samples of a plurality of thermal images of the one or more fruits taken on every day after harvesting, wherein the one or more fruits are from cold storage or room temperature; and these thermal mages are then used to train the deep learning model; and
using the plurality of thermal images to train a deep learning model;
a feature extracting processing unit configured to extract features from the scanned thermal image, wherein a training dataset of the received thermal images is augmented to a pre-trained model via transfer learning, wherein using the transfer learning, the weights of the pre-trained model are updated and are then used to classify a new thermal image data set which includes a thermal image of any new fruit to be tested for edibility in terms of number of days, and wherein features of the thermal image of the new fruit are extracted from convolution and pooling layers of the pre-trained model.
However, Izquierdo, whose invention lies in the field of thermal imaging as applied to food samples, discloses a thermal imaging system for determining shelf life of a fruit, said system comprising:
a thermal imaging device (5) configured to capture thermal images of one or more fruits on receiving the input by the first receiver, wherein said thermal imaging device is configured to convert infrared radiation released by the fruit and surrounding of the fruit into a visible thermal image (Fig. 1-4, 6, 10; Pg. 2, lines 23-27; Pg. 3, lines 4-7 – Izquierdo discloses the use of a thermal imager in determining the thermal distribution of food samples and a use case of applying the method in food and agricultural industries, including determining thermal properties of food and fruit control; furthermore, the thermal imager would inherently capture infra-red radiation released from the fruits surroundings).
It would have been obvious, to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Schwartzer with the sensing of radiation released by the fruit as disclosed by Izquierdo. Unlike conventional near-infrared reflectance (NIR) devices which rely on irradiating a sample and measuring the reflected light, Izquierdo’s modification of using a thermal imager to capture light radiated by the sample, as a result of changes in its composition, instead allows for avoiding errors resulting from any influence incident light would have on the food sample, maintaining its integrity. Such a modification provides the advantage of removing redundancy of measurement information and a less complex thermal sensing apparatus design while also improving the efficiency of the measurement process.
Schwartzer as modified by Izquierdo above does not teach
that the thermal imaging device captures images from all directions,
that the processing unit is configured to ascertain shelf life of the one or more fruits by:
determining the maximum and minimum temperature value of the fruit along with the temperature distribution of an entire fruit to differentiate whether scanned fruit is from cold storage or normal temperature automatically; and
classifying the received thermal images of the one or more fruits into a plurality of classes according to a number of days the fruit will remain edible as particular fruit's shelf-life, wherein classifying the received thermal images is done by:
comparing detailed temperature distribution with pre-defined threshold values also called as weights obtained by a trained model, by applying transfer learning, which is a deep learning technique, on a thermal dataset of a same standard fruit, on the processing unit to determine the shelf life of the fruit, wherein the thermal dataset comprises samples of a plurality of thermal images of the one or more fruits taken on every day after harvesting wherein the one or more fruits are from cold storage or room temperature; and
using the plurality of thermal images to train a deep learning model;
a feature extracting processing unit configured to extract features from the scanned thermal image, wherein a training dataset of the received thermal images is augmented to a pre-trained model via transfer learning, wherein using the transfer learning, the weights of the pre-trained model are updated and are then used to classify a new thermal image data set which includes a thermal image of any new fruit to be tested for edibility in terms of number of days, and wherein features of the thermal image of the new fruit are extracted from convolution and pooling layers of the pre-trained model.
However, Izquierdo discloses a processing unit (6) configured to ascertain shelf life of said one or more fruits based on the received thermal images, wherein said processing unit is configured to ascertain shelf life of the one or more fruits; by:
determining the maximum and minimum temperature value of the fruit along with the temperature distribution of an entire fruit to differentiate whether scanned fruit is from cold storage or normal temperature automatically (Fig. 4 shows a thermal image of a food sample, where a range of temperatures including a maximum and minimum value, is shown using a color-scaled temperature distribution and a corresponding color bar); and
classifying the received thermal images of the one or more fruits into a plurality of classes according to a number of days the fruit will remain edible as particular fruit's shelf-life; wherein classifying the received thermal images is done by (Pg. 2, lines 39-42):
comparing detailed temperature distribution (Pg. 3, lines 4-7) with threshold values, also called as weights, obtained by a trained model on a thermal dataset of a same standard fruit on the processing unit to determine the shelf life of the fruit (Pg. 3, lines 18-23), wherein the thermal dataset comprises samples of a plurality of thermal images of the one or more fruits taken on every day after harvesting, wherein the one or more fruits are from cold storage or room temperature (Pg. 2, line 55 – Pg. 3, line 2 ); and
using the plurality of thermal images to train a deep learning model (Pg. 2, lines 45-52).
It would have been obvious, to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Schwartzer in view of Izquierdo with a device and process comprising the above features as disclosed by Izquierdo where the use of a machine learning model, specifically convolution neural networks (CNN), is a known method in image processing which allows for improved inference and generalization capabilities, increasing the accuracy of the measurement device (Pg. 2, line 63 – Pg. 3, line 2; Pg. 8, lines 1-7).
Schwartzer as modified by Izquierdo above does not explicitly teach using pre-defined threshold values and applying transfer learning on the dataset to determine the shelf life of the fruit;
that the thermal imaging device captures images from all directions,
a feature extracting processing unit configured to extract features from the scanned thermal image, wherein a training dataset of the received thermal images is augmented to a pre-trained model via transfer learning, wherein using the transfer learning, the weights of the pre-trained model are updated and are then used to classify a new thermal image data set which includes a thermal image of any new fruit to be tested for edibility in terms of number of days, and wherein features of the thermal image of the new fruit are extracted from convolution and pooling layers of the pre-trained model.
However, Rai, in the same field of endeavor of systems and methods of monitoring perishable food items, discloses using pre-defined threshold values and applying transfer learning on a dataset to determine the shelf life of the fruit (Fig. 6 – block 606; [0063]; [0066], lines 1-13 – where the pre-trained CNN is interpreted as implying using pre-defined threshold values/weights);
a feature extracting processing unit (500) configured to extract features from the scanned image, wherein a training dataset of the received images is augmented to a pre-trained model via transfer learning (Fig. 5, 6 – block 606; [0063]; [0066], lines 1-13), wherein using the transfer learning, the weights of the pre-trained model are updated and are then used to classify a new image data set which includes a image of any new fruit to be tested for edibility in terms of number of days, and wherein features of the image of the new fruit are extracted from convolution and pooling layers of the pre-trained model (Fig. 5, 6 – block 606; [0063]; [0066], lines 1-13; [0067] – the Examiner is interpreting the convolution and pooling layers as inherent to CNN machine learning architectures; input data, i.e., food images, to the network are understood to include thermal images of food items at a plurality of lifecycle stages).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Schwartzer in view of Izquierdo with a feature extracting processing unit and machine learning model that uses the technique of transfer learning where the pre-trained network weights provide an initialization point for further processing, improving the overall efficiency and accuracy of the classification method by providing a unique, initial visual representation of the food items from low-level to high-level features (Rai: [0063], last five lines). Furthermore, it would have been obvious to one of ordinary skill in the art to use CNNs, as they are the state of the art in image processing machine learning methods, taking into account the spatial structure of images and making the method well adapted for image classification and feature extraction.
Schwartzer as modified by Izquierdo and Rai above does not teach
that the thermal imaging device captures images from all directions,
a motor configured to rotate 360 degrees to assist said thermal imaging device to scan and capture the thermal images of said one or more fruits from all directions.
However, Schwartzer’s thermal imaging device, shown in Fig. 5, appears as a handheld device capable of capturing images from all directions if rotated, to determine the level of ripeness of a fruit; furthermore Hollenbeck, whose invention lies in the field of three-dimensional imaging, discloses
a platform and one or more linear or rotational axes configured move the object relative to the imaging device to assist said imaging device to capture images of said one or more objects from all directions ([0028], lines 1-5; [0028], lines 12-15; [0081], lines 8-12).
It would have been obvious, to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Schwartzer in view of Izquierdo and Rai with a motor configured to rotate 360 degrees to assist said thermal imaging device to capture thermal images from all directions providing the advantage of a faster and more efficient imagining system (Hollenbeck: [0005], lines 1-4) that is able to capture a complete three-dimensional scan of the fruit and its surroundings. Though not explicitly disclosed in Hollenbeck, one of ordinary skill in the art would recognize that a motor configured to rotate 360 degrees to assist said thermal imaging device is functionally equivalent and an obvious alternative to a motorized platform with rotational and translation degrees of freedom which is able to move an object relative to the imaging system.
Claim 3 is the corresponding process claim.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MAHER YAZBACK whose telephone number is (703)756-1456. The examiner can normally be reached Monday - Friday 8:30 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle Iacoletti can be reached at (571)270-5789. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MAHER YAZBACK/Examiner, Art Unit 2877
/MICHELLE M IACOLETTI/Supervisory Patent Examiner, Art Unit 2877