DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-20 are pending for examination in the application filed 10/07/2024.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 10/21/2024 and 02/06/2026 have been considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-2, 7-10 and 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Li (Li, Song, et al. "Assessing fruit hardness in robot hands using electric gripper actuators with tactile sensors." Sensors and Actuators A: Physical 365 (2024): 114843) in view of Aroca (Aroca, Rafael V., et al. "A Wearable Mobile Sensor Platform to Assist Fruit Grading" Sensors (2013): 13(5):6109-6140).
Regarding claim 1, Li teaches a method comprising: generating, by using a vision-based tactile sensor (VBTS) of a device, palpation data corresponding to a deformation detected by the VBTS upon a contact of the VBTS with an object that includes at least one of: a fruit, a legume, or a vegetable ([2.1 Test equipment presentation] The DIGIT sensor is a tactile sensor developed by Mate Inc. A soft elastomer is used to touch an external object. An embedded camera takes a picture of the surface of the elastomer and captures the geometric changes of the surface through a specially designed optical system. [3.3.1 Data collection] Four different types of fruits were involved: tangerine, bananas, persimmons, and kiwifruit, and a total of 480 videos were obtained by performing 120 grabs for each type of fruit, resulting in 480 image sequences. For the kiwifruit ripeness hardness assessment experiment, we numbered 150 experimental kiwifruits and performed 5 grabbing experiments per kiwifruit, collecting a total of 750 videos according to the collection method described above. We can see that the fruits were deformed differently during the pressing and pinching process due to different hardness and surface texture. During the collection process, the live camera and DIGIT capture data at 30hz and 1920 × 1080, and 640 × 480 resolution, respectively. These data give the direct tactile characteristics belonging to each fruit);
generating an input to a machine learning (ML) model based on the palpation data ([2.2.2. Neural network design] In this work, our goal is to be able to use tactile image cues to predict the hardness of fruits. For this purpose, we use a neural network to map image sequences to hardness scale values. As shown in Fig. 5, the input acquired image sequence frames, the features of each image are obtained by convolution using ReLU);
determining an output of the ML model in response to the input, wherein the output indicates a firmness of the object ([2.2.2. Neural network design] The attention mechanism achieves further feature refinement by adding different weights to different positions of the feature map, increasing the weights for favorable features, and decreasing the weights for redundant, irrelevant features. It is then fed to the CapsNet layer and finally regressed on the output hardness values at each time step by affine transformation after the capsule convolution operation…We average the predictions of the last two frames to estimate the hardness values of the fruit);
and presenting, by using a user interface (UI) of the device.
PNG
media_image1.png
418
1426
media_image1.png
Greyscale
Li does not explicitly teach a wearable device; and presenting, by using a user interface (UI) of the wearable device, an indication of the firmness.
Aroca, in the same field of endeavor of measuring fruit firmness, teaches a wearable device;
PNG
media_image2.png
688
905
media_image2.png
Greyscale
and presenting, by using a user interface (UI) of the wearable device, an indication of the firmness ([3. Wearable Sensing Platform Architecture and Implementation] Figure 1 also shows the ez430-Crhonos programmable wrist watch from Texas Instruments, which can be used to easily display numeric information measured by the glove's sensors to the user. [Figure 9.] Top and bottom view of the glove. On the left are finger bending sensors, IMU and USB HUB. On the right there are the optical sensors and finger pressure sensors. [4.1. Overview] when the user touches the fruit, the volume is also computed based on the finger bending sensors, its turgor pressure is measured and an overall quality parameter is computed according to weights for each of these parameters that can be provided by the user. [2.2.2. Mechanical Techniques: Firmness] accurate and non-destructive measurement of fruit firmness, based on cell's turgor pressure, which is known as the pressure difference between cell interior and the barometric pressure).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Li with the teachings of Aroca to use a wearable device because "As we use hands for most of our daily tasks, one interesting solution is to integrate sensors in our hands to build a wearable mobile sensing platform in the form of a glove. The advantage of such platform is that the hands of the user and his attention become free for his task while the glove acts as an intuitive supporting device that provides and collects information about the task being done" [1. Introduction] and present an indication of firmness because "Among several mechanical properties of fruits, firmness is considered one of the most important quality parameters" [2.2.2. Mechanical Techniques: Firmness].
Regarding claim 2, Li and Aroca teach the method of claim 1. Li further teaches wherein the input includes the palpation data, and wherein the method further comprises: displaying, by the UI.
PNG
media_image1.png
418
1426
media_image1.png
Greyscale
Li does not explicitly teach displaying, by the UI, a characterization of the firmness of the fruit, legume, or vegetable.
Aroca, in the same field of endeavor of measuring fruit firmness, teaches displaying, by the UI, a characterization of the firmness of the fruit, legume, or vegetable ([3. Wearable Sensing Platform Architecture and Implementation] Figure 1 also shows the ez430-Crhonos programmable wrist watch from Texas Instruments, which can be used to easily display numeric information measured by the glove's sensors to the user. [Figure 9.] Top and bottom view of the glove. On the left are finger bending sensors, IMU and USB HUB. On the right there are the optical sensors and finger pressure sensors. [4.1. Overview] when the user touches the fruit, the volume is also computed based on the finger bending sensors, its turgor pressure is measured and an overall quality parameter is computed according to weights for each of these parameters that can be provided by the user. [2.2.2. Mechanical Techniques: Firmness] accurate and non-destructive measurement of fruit firmness, based on cell's turgor pressure, which is known as the pressure difference between cell interior and the barometric pressure).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Li with the teachings of Aroca to display a characterization of firmness because "Among several mechanical properties of fruits, firmness is considered one of the most important quality parameters" [2.2.2. Mechanical Techniques: Firmness].
Regarding claim 7, Li teaches a device comprising: a vision-based tactile sensor (VBTS) configured to generate palpation data upon contact of the VBTS with an object that includes at least one of a fruit, legume, or vegetable ([2.1 Test equipment presentation] The DIGIT sensor is a tactile sensor developed by Mate Inc. A soft elastomer is used to touch an external object. An embedded camera takes a picture of the surface of the elastomer and captures the geometric changes of the surface through a specially designed optical system. [3.3.1 Data collection] Four different types of fruits were involved: tangerine, bananas, persimmons, and kiwifruit, and a total of 480 videos were obtained by performing 120 grabs for each type of fruit, resulting in 480 image sequences. For the kiwifruit ripeness hardness assessment experiment, we numbered 150 experimental kiwifruits and performed 5 grabbing experiments per kiwifruit, collecting a total of 750 videos according to the collection method described above. We can see that the fruits were deformed differently during the pressing and pinching process due to different hardness and surface texture. During the collection process, the live camera and DIGIT capture data at 30hz and 1920 × 1080, and 640 × 480 resolution, respectively. These data give the direct tactile characteristics belonging to each fruit);
and a user interface (UI),
PNG
media_image1.png
418
1426
media_image1.png
Greyscale
the indication generated by a machine learning model (ML) based on the palpation data ([2.2.2. Neural network design] In this work, our goal is to be able to use tactile image cues to predict the hardness of fruits. For this purpose, we use a neural network to map image sequences to hardness scale values. As shown in Fig. 5, the input acquired image sequence frames, the features of each image are obtained by convolution using ReLU…The attention mechanism achieves further feature refinement by adding different weights to different positions of the feature map, increasing the weights for favorable features, and decreasing the weights for redundant, irrelevant features. It is then fed to the CapsNet layer and finally regressed on the output hardness values at each time step by affine transformation after the capsule convolution operation…We average the predictions of the last two frames to estimate the hardness values of the fruit).
Li does not explicitly teach a first wearable interface; a sensor attached to the first wearable interface; a second wearable interface; and a user interface (UI) attached to the second wearable interface and configured to present an indication of a firmness of the object.
Aroca, in the same field of endeavor of measuring fruit firmness, teaches a first wearable interface; a sensor attached to the first wearable interface; a second wearable interface; and a user interface (UI) attached to the second wearable interface and configured to present an indication of firmness of the object ([1. Introduction] We propose a novel mobile sensing platform mounted on a glove that integrates several sensors, such as touch pressure, imaging, inertial measurements, localization and a Radio Frequency Identification (RFID) reader. [3. Wearable Sensing Platform Architecture and Implementation] Figure 1 also shows the ez430-Crhonos programmable wrist watch from Texas Instruments, which can be used to easily display numeric information measured by the glove's sensors to the user. [4.1. Overview] when the user touches the fruit, the volume is also computed based on the finger bending sensors, its turgor pressure is measured and an overall quality parameter is computed according to weights for each of these parameters that can be provided by the user. [2.2.2. Mechanical Techniques: Firmness] accurate and non-destructive measurement of fruit firmness, based on cell's turgor pressure, which is known as the pressure difference between cell interior and the barometric pressure).
PNG
media_image2.png
688
905
media_image2.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Li with the teachings of Aroca to use a first wearable interface attached to a sensor and a second wearable interface attached to a user interface because "As we use hands for most of our daily tasks, one interesting solution is to integrate sensors in our hands to build a wearable mobile sensing platform in the form of a glove. The advantage of such platform is that the hands of the user and his attention become free for his task while the glove acts as an intuitive supporting device that provides and collects information about the task being done" [1. Introduction] and "to show information about the glove's sensors to the user. A possible use is to allow users to view quantitative information about the sensors" [3.3. Other Sensors and Actuators] and present an indication of firmness because "Among several mechanical properties of fruits, firmness is considered one of the most important quality parameters" [2.2.2. Mechanical Techniques: Firmness].
Regarding claim 8, Li and Aroca teach the device of claim 7. Li further teaches processing circuitry configured to send the palpation data to a system executing the ML model, wherein the processing circuitry is configured to receive the firmness as an output of the ML model ([2.2.2. Neural network design] In this work, our goal is to be able to use tactile image cues to predict the hardness of fruits. For this purpose, we use a neural network to map image sequences to hardness scale values. As shown in Fig. 5, the input acquired image sequence frames, the features of each image are obtained by convolution using ReLU…The attention mechanism achieves further feature refinement by adding different weights to different positions of the feature map, increasing the weights for favorable features, and decreasing the weights for redundant, irrelevant features. It is then fed to the CapsNet layer and finally regressed on the output hardness values at each time step by affine transformation after the capsule convolution operation…We average the predictions of the last two frames to estimate the hardness values of the fruit).
PNG
media_image3.png
416
703
media_image3.png
Greyscale
Li does not explicitly teach processing circuitry attached to the second wearable interface; and cause the presentation of the firmness at the UI.
Aroca, in the same field of endeavor of measuring fruit firmness, teaches processing circuitry attached to the second wearable interface; and cause the presentation of the firmness at the UI ([3.3. Other Sensors and Actuators] A programmable wrist watch such as the ez430 Chronos can also be used to show information about the glove's sensors to the user. A possible use is to allow users to view quantitative information about the sensors. [4.1. Overview] when the user touches the fruit, the volume is also computed based on the finger bending sensors, its turgor pressure is measured and an overall quality parameter is computed according to weights for each of these parameters that can be provided by the user. [2.2.2. Mechanical Techniques: Firmness] accurate and non-destructive measurement of fruit firmness, based on cell's turgor pressure, which is known as the pressure difference between cell interior and the barometric pressure).
PNG
media_image4.png
282
213
media_image4.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Li with the teachings of Aroca to use processing circuitry attached to the second wearable interface and cause a presentation of firmness at the UI "to show information about the glove's sensors to the user. A possible use is to allow users to view quantitative information about the sensors" [3.3. Other Sensors and Actuators] and "Among several mechanical properties of fruits, firmness is considered one of the most important quality parameters" [2.2.2. Mechanical Techniques: Firmness].
Regarding claim 9, Li and Aroca teach the device of claim 7. Li further teaches processing circuitry configured to execute the ML model ([2.2.2. Neural network design] In this work, our goal is to be able to use tactile image cues to predict the hardness of fruits. For this purpose, we use a neural network to map image sequences to hardness scale values. As shown in Fig. 5, the input acquired image sequence frames, the features of each image are obtained by convolution using ReLU…The attention mechanism achieves further feature refinement by adding different weights to different positions of the feature map, increasing the weights for favorable features, and decreasing the weights for redundant, irrelevant features. It is then fed to the CapsNet layer and finally regressed on the output hardness values at each time step by affine transformation after the capsule convolution operation…We average the predictions of the last two frames to estimate the hardness values of the fruit).
PNG
media_image3.png
416
703
media_image3.png
Greyscale
Li does not explicitly teach processing circuitry attached to the second wearable interface.
Aroca, in the same field of endeavor of measuring fruit firmness, teaches processing circuitry attached to the second wearable interface ([3.3. Other Sensors and Actuators] A programmable wrist watch such as the ez430 Chronos can also be used to show information about the glove's sensors to the user. A possible use is to allow users to view quantitative information about the sensors).
PNG
media_image4.png
282
213
media_image4.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Li with the teachings of Aroca to use processing circuitry attached to the second wearable interface "to show information about the glove's sensors to the user. A possible use is to allow users to view quantitative information about the sensors" [3.3. Other Sensors and Actuators].
Regarding claim 10, Li and Aroca teach the device of claim 7. Li teaches the VBTS ([2.1 Test equipment presentation] The DIGIT sensor is a tactile sensor developed by Mate Inc. A soft elastomer is used to touch an external object. An embedded camera takes a picture of the surface of the elastomer and captures the geometric changes of the surface through a specially designed optical system).
Li does not explicitly teach wherein the first wearable interface further comprises: a housing configured to receive a thumb at a first end, wherein the housing is configured to receive the sensor at a second end; and at least one camera.
Aroca, in the same field of endeavor of measuring fruit firmness, teaches wherein the first wearable interface further comprises: a housing configured to receive a thumb at a first end, wherein the housing is configured to receive the sensor at a second end; and at least one camera ([3. Wearable Sensing Platform Architecture and Implementation] This HUB connects 2 USB cameras mounted in the palm of the hand, a RFID reader, also shown in Figure 1, and a 915 MHz radio to exchange information with a programmable wrist watch…All the sensors fixed to glove are connected to 8-bit microcontrollers (AtMega 328)).
PNG
media_image5.png
292
842
media_image5.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Li with the teachings of Aroca to include a housing to receive a thumb at a first end and the sensor at the second end, and a camera because "As we use hands for most of our daily tasks, one interesting solution is to integrate sensors in our hands to build a wearable mobile sensing platform in the form of a glove. The advantage of such platform is that the hands of the user and his attention become free for his task while the glove acts as an intuitive supporting device that provides and collects information about the task being done" [1. Introduction].
Regarding claim 12, Li and Aroca teach the device of claim 7. Li does not explicitly teach wherein the first wearable interface and the second wearable interface are linked together by a coupler selected from a group comprising: a wired connection, a wireless connection, a fabric, a tether, or combinations thereof.
Aroca, in the same field of endeavor of measuring fruit firmness, teaches wherein the first wearable interface and the second wearable interface are linked together by a coupler selected from a group comprising: a wired connection, a wireless connection, a fabric, a tether, or combinations thereof.
PNG
media_image2.png
688
905
media_image2.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Li with the teachings of Aroca to connect the first and second wearable interfaces with a wireless coupler "to exchange information" [3. Wearable Sensing Platform Architecture and Implementation].
Regarding claim 13, Li and Aroca teach the device of claim 7. Li further teaches wherein the VBTS includes a surface configured to contact the object, and wherein the surface is an elastomer configured to deform when placed in contact with the object ([2.1. Test equipment presentation] The DIGIT sensor is a tactile sensor developed by Mate Inc. A soft elastomer is used to touch an external object. An embedded camera takes a picture of the surface of the elastomer and captures the geometric changes of the surface through a specially designed optical system. The physical and schematic diagrams are shown in Fig. 1: The sensor is similar to the skin sensing of human fingertips in that when the sensor is pressed on an object, it infers touch information from the deformation of the soft tissue, resulting in differences in surface curvature and contact force. For example, when a softer object is pressed, the soft object deforms more and the ridges flatten out, producing a smoother surface).
Claims 3 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Aroca and Alambeigi (US20250031968A1).
Regarding claim 3, Li and Aroca teach the method of claim 1. Li does not explicitly teach wherein the palpation data is generated based on the VBTS being in contact with the object without the object being removed from a parent plant to which object is attached and without a damage to the object.
Alambeigi, in the same field of endeavor of fruit sensing analysis, teaches wherein the palpation data is generated based on the VBTS being in contact with the object without the object being removed from a parent plant to which object is attached and without a damage to the object ([0007] Besides the aforementioned TS devices, Vision-based Tactile Sensors (VTSs) have also recently been developed to enhance tactile perception via high-resolution visual information…Particularly, VTSs can provide qualitative 3D visual image reconstruction and localization of the interacting rigid or deformable objects by capturing very small deformations of an elastic gel layer that directly interacts with the objects' surface. [0083] In some embodiments, the measured object comprises a fruit. For example, the system 100 can be utilized in a farming application such as harvesting by taking a measurement of the fruit to see if is ripe enough to pick).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Li with the teachings of Alambeigi to generate palpitation data based on the VBTS being in contact with the object without it being removed from the plant or damaged for "real-time post processing and analysis of the high-resolution TS information provided by this sensor" [0007] "utilized in a farming application such as harvesting by taking a measurement of the fruit to see if is ripe enough to pick" [0083].
Claims 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Aroca and Rakshit (US20250200824A1).
Regarding claim 4, Li and Aroca teach the method of claim 1. Li does not explicitly teach wherein the ML model is executed on a system remote from the wearable device, wherein the method further includes: sending the palpation data as the input to the system; and receiving the output of the ML model from the system.
Rakshit, in the same field of endeavor of object deformation analysis, teaches wherein the ML model is executed on a system remote from the wearable device ([0113] The computer 801 may take the form of a desktop computer, laptop computer, tablet computer, smart phone, smart watch or other wearable computer…accessing a network or querying a database, such as the remote database 830. As is well understood in the art of computer technology, and depending upon the technology, performance of a computer-implemented method may be distributed among multiple computers and/or between multiple locations. [0069] In some embodiments, object deformation analysis device 102 can utilize machine learning and/or deep learning, where algorithms or models can be generated by performing supervised, unsupervised, or semi-supervised training on historical data inputs and/or historical features),
wherein the method further includes: sending the palpation data as the input to the system; and receiving the output of the ML model from the system ([0067] In a software implementation, when a machine learning model is referred to as receiving an input, executed, and/or as generating an output or predication, a computer system process, such as object deformation analysis device 102, executing a machine learning algorithm applies the model artifact against the input to generate a predicted output. [0054] In embodiments, communication between shopping carts/trolleys on the floor allows the sharing of movement patterns, dynamic load information, and relative positions. The system predicts the mobility path of the trolley based on these inputs and proactively illustrates potential deformations that may occur due to the anticipated dynamic load paths).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Li with the teachings of Rakshit to execute the ML model remotely from the wearable device because it "provides on-demand availability of computer system resources and/or other computer capabilities, especially data storage (cloud storage) and computing power, without direct active management by the user" [0124] and send the palpitation data as input to the ML system and receive an output because "In the context of vegetables and fruits, the weight of the objects above can cause the lower ones to become flattened, bruised, or even crushed. Similarly, packaged products can experience compression deformation if they are stacked too high or if heavy objects are placed on top of them. This type of deformation can lead to not only physical damage but also spoilage of delicate objects, reducing their quality and shelf life" [0003].
Regarding claim 5, Li and Aroca teach the method of claim 1. Li does not explicitly teach executing, locally on the wearable device, the ML model.
Rakshit, in the same field of endeavor of object deformation analysis, teaches executing, locally on the wearable device, the ML model ([0113] The computer 801 may take the form of a desktop computer, laptop computer, tablet computer, smart phone, smart watch or other wearable computer…in this presentation of the computing environment 800, detailed discussion is focused on a single computer, specifically the computer 801. [0069] In some embodiments, object deformation analysis device 102 can utilize machine learning and/or deep learning, where algorithms or models can be generated by performing supervised, unsupervised, or semi-supervised training on historical data inputs and/or historical features).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Li with the teachings of Rakshit to execute the ML model locally so that it is "a single computer" [0113].
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Aroca and Sarvazyan (US5706815A).
Regarding claim 6, Li and Aroca teach the method of claim 1. Li does not explicitly teach wherein a time interval between generating the palpation data and presenting the indication is in a range between 0.1 and 20.0 seconds.
Sarvazyan, in the same field of endeavor of firmness measurement, teaches wherein a time interval between generating the palpation data and presenting the indication is in a range between 0.1 and 20.0 seconds ([col. 2 ln. 11-14] For example, the ripeness, or freshness of a food product can often be determined by analyzing the mechanical properties of the tissue of the food product, such as elasticity, or hardness. [Abstract] A device and method for measuring the anisotropic mechanical properties of tissue is provided. The device comprises a flexural resonator driven by a transducer to oscillate at a plurality of angles of oscillation while in contact with a tissue to be analyzed…Based on these values and the free state resonance frequency of the oscillating flexural resonator, anisotropic mechanical properties of the tissue are derived…and displaying these values in a useful manner. [col. 13 ln. 25-27] In a preferred embodiment, the flowchart is implemented through software which is stored in the memory unit 33 and executed by the microprocessor 30 of the electrical module. Using the microprocessor embodiment, a complete analysis can be made in approximately three seconds).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Li with the teachings of Sarvazyan to generate the palpitation data and present the indication between 0.1 and 20.0 seconds "Using the microprocessor embodiment" [col. 13 ln. 26] because "The display 31 is controlled by and receives data from the microprocessor 30 which determines the data to be displayed and the presentation of that data…In a preferred embodiment, the display unit 31 will show a radial graph on which the anisotropic mechanical properties are plotted corresponding to each angle of oscillation phi at which the properties were determined. Other graphical representations can also be used, but the radial graph is often the most helpful in presenting the variations in the properties in different directions" [col. 9 ln. 39-60].
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Aroca and Texas Instruments (Texas Instruments. (2009). eZ430-Chronos™ Development Tool User's Guide. https://www.ti.com/lit/ug/slau292g/slau292g.pdf?ts=1782882359378).
Regarding claim 11, Li and Aroca teach the device of claim 7. Li does not explicitly teach wherein the second wearable interface further comprises: a housing configured to couple to an appendage; a screen configured to display a graphical user interface (GUI) and at least partially disposed within the second wearable interface, wherein the UI includes the GUI; and wherein the second wearable interface is configured to at least partially house processing circuitry and a transceiver.
Aroca, in the same field of endeavor of measuring fruit firmness, teaches wherein the second wearable interface further comprises: a housing configured to couple to an appendage; a screen configured to display a graphical user interface (GUI) and at least partially disposed within the second wearable interface, wherein the UI includes the GUI ([3.3. Other Sensors and Actuators] A programmable wrist watch such as the ez430 Chronos can also be used to show information about the glove's sensors to the user. A possible use is to allow users to view quantitative information about the sensors).
PNG
media_image4.png
282
213
media_image4.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Li with the teachings of Aroca for the second wearable interface to comprise housing configured to couple to an appendage and display a GUI because "the ez430-Crhonos programmable wrist watch from Texas Instruments, [can] be used to easily display numeric information measured by the glove's sensors to the user" [3. Wearable Sensing Platform Architecture and Implementation].
Texas Instruments teaches wherein the second wearable interface is configured to at least partially house processing circuitry and a transceiver ([1.2 eZ430-Chronos Features] Sports watch development kit based on the CC430F6137, an MSP430™ microcontroller with integrated sub-1-GHz wireless transceiver).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the method of Li with the teachings of Texas Instruments to partially house processing circuitry and a transceiver "to act as a central hub for nearby wireless sensors" [1.1 Overview].
Claims 14-15, 17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Sawada (JP2022111054A).
Regarding claim 14, Li teaches operations comprising: generating, by using a vision-based tactile sensor (VBTS), palpation data corresponding to a deformation detected by the VBTS upon a contact of the VBTS with an object ([2.1 Test equipment presentation] The DIGIT sensor is a tactile sensor developed by Mate Inc. A soft elastomer is used to touch an external object. An embedded camera takes a picture of the surface of the elastomer and captures the geometric changes of the surface through a specially designed optical system. [3.3.1 Data collection] Four different types of fruits were involved: tangerine, bananas, persimmons, and kiwifruit, and a total of 480 videos were obtained by performing 120 grabs for each type of fruit, resulting in 480 image sequences. For the kiwifruit ripeness hardness assessment experiment, we numbered 150 experimental kiwifruits and performed 5 grabbing experiments per kiwifruit, collecting a total of 750 videos according to the collection method described above. We can see that the fruits were deformed differently during the pressing and pinching process due to different hardness and surface texture. During the collection process, the live camera and DIGIT capture data at 30hz and 1920 × 1080, and 640 × 480 resolution, respectively. These data give the direct tactile characteristics belonging to each fruit);
generating an input to a machine learning (ML) model based on the palpation data ([2.2.2. Neural network design] In this work, our goal is to be able to use tactile image cues to predict the hardness of fruits. For this purpose, we use a neural network to map image sequences to hardness scale values. As shown in Fig. 5, the input acquired image sequence frames, the features of each image are obtained by convolution using ReLU);
determining an output of the ML model in response to the input, wherein the output indicates a firmness of the object ([2.2.2. Neural network design] The attention mechanism achieves further feature refinement by adding different weights to different positions of the feature map, increasing the weights for favorable features, and decreasing the weights for redundant, irrelevant features. It is then fed to the CapsNet layer and finally regressed on the output hardness values at each time step by affine transformation after the capsule convolution operation…We average the predictions of the last two frames to estimate the hardness values of the fruit);
and causing a presentation at a user interface.
PNG
media_image1.png
418
1426
media_image1.png
Greyscale
Li does not explicitly teach a non-transitory computer readable medium having stored thereon computer-readable instructions that, when executed by a processor, cause the processor to perform operations comprising: causing a presentation of an indication of the firmness at a user interface (UI).
Sawada, in the same field of endeavor of fruit evaluation, teaches a non-transitory computer readable medium having stored thereon computer-readable instructions that, when executed by a processor, cause the processor to perform operations comprising: causing a presentation of an indication of the firmness at a user interface (UI) ([pg. 3 para. 3] FIG. 2 shows a specific configuration example of the estimation device 2. The estimating device 2 performs wired or wireless communication with a control unit 24 for controlling the entire estimating device 2 and an operating unit 25 for inputting various control commands via operation buttons, a keyboard, or the like. A communication unit 26 for searching, an estimating unit 27 for making various judgments, and a storage unit 28, represented by a hard disk, for storing a program for performing a search to be executed are connected to the internal bus 21, respectively. Further, the internal bus 21 is connected with a display unit 23 as a monitor for actually displaying information. [pg. 4 para. 3-4] The quality of the apples may be judged based on previous experience by the evaluator, or may be judged by actually tasting the apples. In such a case, multiple testers who taste the apples evaluate each item such as sweetness, sourness, softness, hardness, aroma, texture, bitterness, etc. on multiple stages, and the results are statistically evaluated. It may be analyzed systematically and used as a quality evaluation value…In the example of FIG. 3, it is assumed that the input data are, for example, reference appearance information P01 to P03. Such reference appearance information P01 to P03 as input data are linked to the quality of apples as output. In this output, the apple quality is displayed as the output solution).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify Li with the teachings of Sawada to use a non-transitory computer readable medium "for storing a program for performing a search to be executed" [pg. 3 para. 3] and a presentation of an indication of firmness because "In this way, it is possible to search for the most suitable apple quality from the newly acquired appearance information and display it to the user. By looking at the results of this search, users, namely apple growers, sellers, and distributors, can sort apples based on the quality of the searched apples, and can predict the taste of apples. In addition, you can determine the price of apples. Moreover, since the quality of these products can be predicted through non-destructive inspection without touching the apples, the apples will not be damaged, and the yield can be improved" [pg. 5 para. 4].
Regarding claim 15, Li and Sawada teach the medium of claim 14. Li further teaches wherein the object is a fruit or vegetable ([Abstract] Fruit hardness is an indispensable attribute for appraising fruit quality, with notable implications for nondestructive robotic grasping, ripeness determination, and classification. This article introduces an tactile predictive recognition approach that leverages an adaptive capsule network to assess fruit hardness through manual interaction).
Li does not explicitly teach wherein the operations further comprise: determining a harvest time of the object based on the firmness, wherein the harvest time is an estimated time predicted to elapse before the object is ripe; and presenting, using the UI, the harvest time.
Sawada, in the same field of endeavor of fruit evaluation, teaches wherein the operations further comprise: determining a harvest time of the object based on the firmness, wherein the harvest time is an estimated time predicted to elapse before the object is ripe ([pg. 13 para. 6] Further, hardness information for reference and hardness information may be used instead of sugar content information for reference and sugar content information. The hardness information for reference and the hardness information are information relating to the hardness of the apple, and may be measured with a general hardness tester or evaluated by a texture test, for example. [pg. 4 para. 3-4] The quality of the apples may be judged based on previous experience by the evaluator, or may be judged by actually tasting the apples. In such a case, multiple testers who taste the apples evaluate each item such as sweetness, sourness, softness, hardness, aroma, texture, bitterness, etc. on multiple stages, and the results are statistically evaluated. It may be analyzed systematically and used as a quality evaluation value…In the example of FIG. 3, it is assumed that the input data are, for example, reference appearance information P01 to P03. Such reference appearance information P01 to P03 as input data are linked to the quality of apples as output. In this output, the apple quality is displayed as the output solution. [pg. 5 para. 5] At this time, the quality of each apple may be associated with harvest time information. Harvest time information here is information indicating when to harvest, and includes, for example, when to harvest, such as whether to harvest now, two days later, four days later, one week later, and the like);
and presenting, using the UI, the harvest time ([pg. 5 para. 5] Then, after estimating the quality of the apples, the linked harvest time status is displayed. This allows the user to know when to harvest apples before flowering).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify Li with the teachings of Sawada to determine a harvest time based on the firmness and present the harvest time because "the quality of the apple as the obtained search solution or the harvest time information linked thereto may be displayed on the display of the HMD. In such a case, so-called AR (Augmented Reality) technology may be used to display information on proposed flowers to be picked on a transparent display. As a result, in the image of the apples before thinning that are visually recognized on the screen, which apples should be harvested immediately, including the suitability of harvesting, can be displayed in a transparent state, and the convenience of the user's work can be displayed. In addition to displaying these information on the display in a transparent state, the user may be notified" [pg. 5 para. 7].
Regarding claim 17, Li and Sawada teach the medium of claim 14. Li further teaches wherein the palpation data includes one or more tactile images ([2.1 Test equipment presentation] The DIGIT sensor is a tactile sensor developed by Mate Inc. A soft elastomer is used to touch an external object. An embedded camera takes a picture of the surface of the elastomer and captures the geometric changes of the surface through a specially designed optical system);
and wherein the operations further comprise: receiving, from the VBTS, a first tactile image when VBTS initially contacts the object; receiving, from the VBTS, additional tactile images while VBTS contacts the object during a time interval, wherein the one or more tactile images includes the first tactile image and the additional tactile images ([3.3.1. Data collection] Once the jaws are closed, the tactile sensors at the fingertips record tactile data. The motorized jaws used can be set to close with a slow and constant force until the gripping force reaches a threshold. The jaw speed was randomly selected between 4 and 6 mm/s, and the gripping force was set to 15 N/m. We collected our own dataset in different types of fruit hardness classification. Four different types of fruits were involved: tangerine, bananas, persimmons, and kiwifruit, and a total of 480 videos were obtained by performing 120 grabs for each type of fruit, resulting in 480 image sequences);
PNG
media_image6.png
436
690
media_image6.png
Greyscale
and causing a token operation to be performed by the ML model on the one or more tactile images to generate firmness data associated with the firmness of the object ([2.2.2. Choosing input sequences] The acquisition video recording starts when the tactile sensory layer comes in contact with the fruit. As the press continues, the intensity change of pixels around the press increases, and the intensity change should peak in the last frame, and the video acquisition recording ends. Here, eight image sequences in the press video are taken for analysis. After acquiring the first and last frames of the video data, the remaining six frames are evenly distributed in the middle. These eight frames are a uniformly changing process to ensure consistency in the acquisition interaction conditions for each press. Image sequences were first grayed out to ignore the effect of fruit color and then processed separately to obtain depth map sequence images and mask sequence images. As shown in Fig. 4(c)(d), these images contain rich contact features, which are then fed into the TSACN (Tactile Self-Attentive Capsule Network) architecture for analysis to assess hardness).
Regarding claim 19, Li and Sawada teach the medium of claim 17. Li further teaches wherein the operations further comprise: causing training of the ML model using the one or more tactile images;
PNG
media_image7.png
794
919
media_image7.png
Greyscale
and in response to the training, updating the firmness data ([3.3.3. Training] During network training, we used MSELoss as a loss function to minimize to penalize the difference between the predicted hardness values and the ground truth values, and the training process was performed using the Adam optimizer with a learning rate of 0.0005, and the model was implemented on the PyTorch platform and trained on an NVIDIA RTX3090 server).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Sawada and Takahashi (JP2023097192A).
Regarding claim 16, Li and Sawada teach the medium of claim 15. Li does not explicitly teach wherein the operations further comprise: storing the harvest time in a memory; and in response to comparing the harvest time to a time threshold, transmitting a notification to a client device.
Takahashi, in the same field of endeavor of harvesting analysis, teaches wherein the operations further comprise: storing the harvest time in a memory ([pg. 10 para. 2] FIG. 8 is a diagram showing an example of the contents of the growth prediction information 374. As shown in FIG. In the growth prediction information 374, for example, an individual ID is associated with position information, an appropriate harvest time, and a stock weight. The appropriate harvest time is information about the appropriate harvest time for each strain derived from, for example, a growth model, and information about date and time and period is stored);
and in response to comparing the harvest time to a time threshold, transmitting a notification to a client device ([pg. 11 para. 4-5] FIG. 12 is a diagram for explaining notifying the worker P of information on crops suitable for harvest. The example of FIG. 12 shows a state in which broccoli BR1 to BR3 are present in field F as an example of crops. In this case, the information providing unit 270 acquires harvest prediction information for each individual broccoli BR1 to BR3 from the management server 300, and based on the acquired information, the position (for example, latitude , longitude) and the position of the worker P (for example, latitude and longitude), and when the relative distance is within a predetermined distance, the audio output unit 264 outputs audio…In addition, the information providing unit 270 may change the content of the notification according to the number of days until the appropriate harvest time for the individual crop. In this case, the information providing unit 270 increases the volume or changes the sound to a rhythmic sound as the number of days approaches, and outputs the sound…Further, the information providing section 270 may display information indicating that it is the proper harvest time on the display section 262 instead of (or in addition to) outputting the sound…In the example of FIG. 13, as the additional information IN10, character information indicating that the broccoli BR2 is in the right time to harvest (for example, "It is the right time to harvest") is displayed)
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify Li with the teachings of Takahashi to store the harvest time and transmit a notification in response to comparing the harvest time to a threshold because "by providing information on the appropriate harvest period for each crop in the field, for example, when the harvest date is input, the crops suitable for harvest on that day can be picked up and the number and the like can be displayed" [pg. 14 para. 2].
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Sawada and Robinson (US5918266A).
Regarding claim 18, Li and Sawada teach the medium of claim 17. Li further teaches identifying, based at least partially on the one or more tactile images, a type of the object; in response to identifying the type, recalculating the firmness data ([2.3 System setup and workflow] During the grasping process, tactile data is collected to identify the hardness of the fruit. The robot distinguishes the type and ripeness of the fruit based on the identified hardness. [3.3.2. Training] We evaluated the effectiveness of TSACN by training the fruit data separately. For each type of data, we used average normalization and random processing of the data. A 10-fold cross-validation method was used to split the data into training and test data. During network training, we used MSELoss as a loss function to minimize to penalize the difference between the predicted hardness values and the ground truth values, and the training process was performed using the Adam optimizer with a learning rate of 0.0005, and the model was implemented on the PyTorch platform and trained on an NVIDIA RTX3090 server).
Li does not explicitly teach presenting, using the UI, the type and the recalculated firmness data.
Robinson, in the same field of endeavor of fruit firmness evaluation, teaches presenting, using the UI, the type and the recalculated firmness data ([col. 6 ln. 55 - col. 7 ln. 13] The volume change can be directly displayed by the inventive apparatus, which would allow one either to calculate a firmness index directly or to look up the firmness of the object(s) under test in a table which lists firmness as a function of firmness index. Alternately, the displacement signal can be supplied in analog or digital form to a firmness indicating device, suitably calibrated to transform the supplied signal into a display indicative of the firmness measurement. Automatic means such as a digital computer can also be used by supplying the displacement signal in a form suitable for input into the computer (e.g. a digital signal). The computer can then automatically perform the calculations necessary to compute firmness (for fruit firmness would correlate to the measured firmness in a known way, perhaps determined empirically-for each particular type of fruit) and display it in a suitable fashion, or control automatic processing and/or packing equipment. [col. 7 ln. 5-13] This device can be employed as a fixed unit for firmness testing in a packing house or a factory. In a packing house it can be used to sort fruits and other food products on line during the sorting and packing process. It can test the firmness of the fruit on line, correlate the measured firmness index with firmness of the fruit (or some other predetermined property of the fruit or quality parameter), and then provide an input to devices or systems which place the fruit in a pre selected location based on its firmness).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify Li with the teachings of Robinson to present the type and recalculated firmness "to measure the firmness of objects during the packing process which will allow separation of objects based on that firmness" [col. 1 ln. 54-56].
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Li in view of Sawada, Aroca, and Nakauchi (JP2006226775A).
Regarding claim 20, Li and Sawada teach the medium of claim 17. Li does not explicitly teach wherein the operations further comprise: receiving, from the UI on a wearable device, a selection of a type of the object; in response to receiving the type of the object, recalculating the firmness; and presenting, using the UI, the recalculated firmness.
Aroca, in the same field of endeavor of measuring fruit firmness, teaches a wearable device.
PNG
media_image2.png
688
905
media_image2.png
Greyscale
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify Li with the teachings of Aroca to use a wearable device because "As we use hands for most of our daily tasks, one interesting solution is to integrate sensors in our hands to build a wearable mobile sensing platform in the form of a glove. The advantage of such platform is that the hands of the user and his attention become free for his task while the glove acts as an intuitive supporting device that provides and collects information about the task being done" [1. Introduction].
Nakauchi, in the same field of endeavor of fruit firmness evolution, teaches wherein the operations further comprise: receiving, from the UI on a device, a selection of a type of the object ([pg. 9 para. 8] When the processing is started, first, the type of melon to be measured (red meat melon or green melon) is specified from the setting of the kind selection switch (S10));
in response to receiving the type of the object, recalculating the firmness ([pg. 9 para. 8] Based on the results of cultivar identification and cultivation period identification, each of the four sugar content calibration curve databases, the four hardness calibration curve databases, and the four eating time specification databases is used for each subsequent calculation. Each is selected (S30), and the database specifying information is recorded in the RAM (S40). [pg. 10 para. 2] Next, the sugar content and hardness are calculated based on the light detection signals R0 to R4 acquired in S60 to S100, the sugar content calibration curve database, and the hardness calibration curve database (S110, S120). This calculation is executed based on the following equation. Each calibration curve database is obtained in advance);
and presenting, using the UI, the recalculated firmness ([pg. 7-8] When displaying the hardness, a display method such as “unripe”, “appropriate maturity”, or “overripe” may be used. The number of days until eating may be fixed as a message to be displayed depending on where the hardness falls within the above three-level evaluation).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify Li with the teachings of Nakauchi to receive a selection of the type of object, recalculate the firmness based on the type, and display the firmness because "In melon, sugar content and hardness (maturity) are important among the taste components, and these can be accurately evaluated by the absorbance at the above-mentioned wavelength. And the melon purchaser wants to know the best time to eat after many days. Therefore, in the melon taste component evaluation apparatus, not only simply inform the measurement value of sugar content and hardness, it is also possible to provide the number of days until eating as information with the above-mentioned configurations (t) to (v), It meets the demands of consumers" [pg. 7 para. 15] and "in the case of melon, although it is the same kind of fruit, the absorbance data also depends on the difference in the color of the flesh such as red melon and green melon, or the difference in the cultivation and harvesting timing such as spring melon and summer melon. The correlation with sugar content etc. may be different" [pg. 2 para. 6].
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Prussia (US5372030A) teaches a non-destructive fruit firmness measurement device.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jacqueline R Zak whose telephone number is (571)272-4077. The examiner can normally be reached M-F 9-5. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACQUELINE R ZAK/Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666
Prosecution Insights