DETAILED ACTION
Response to Arguments
The amendments filed 7/7/2026 have been entered and made of record.
The Applicant's amendments and arguments filed 7/7/2026 have been considered but are moot in view of the new ground(s) of rejection because the Applicant has amended at least independent claim 1., the amended claims are still rejected Under 35 U.S.C. § 103 as being unpatentable over Patch, in view of newly found reference Dick (US 20190166823 A1), and further in view of YU (CN 114332621 A):
Rejections of Claim 1 Under 35 U.S.C. § 103
3.1, regarding to amended limitation “wherein the capture zone includes a capture surface on which the insect is maintained following transportation into the capture zone, the capture surface being configured to absorb infrared light, and wherein the capture surface is formed of a textured material configured to absorb infrared light”, which is similar to previous claims 4-6, as discussed in the last Office Action of 4/8/2026,
Patch discloses wherein the capture zone includes a capture surface on which the insect is maintained following transportation into the capture zone, the capture surface being configured to absorb infrared light, and wherein the capture surface is formed of a textured material configured to absorb infrared light (see Patch: e.g., -- the elongated tube 114 could be a cuboid shape. However, the elongated tube 114 could be any other suitable shape (e.g., a cubic shape, a cylindrical shape, etc.). In a non-limiting example, the size of the insect entrance of the elongated tube 114 can be designed for an insect to move in a limited and predicted way. The insect entrance of the elongated tube 114 can be a part of the elongated tube 114, which is connected to the first opening 110. An imaging device can record one or more visual images (e.g., photograph, film, video, or any other suitable image) of an expected position of the insect when the insect moves through the elongated tube 114. It should be appreciated that the size of the insect entrance of the elongated tube 114 can be big enough to accept multiple insects at the same time. In another example, the size of the insect entrance of the elongated tube 114 can be designed based on the sizes of insects that the user wants to analyze. Movement and positioning of an insect can also be influenced by alternative surface composition within the insect guide tube 108. For example, one inner surface of the elongated tube 114 can include a rough or adhesive surface for an insect to walk on the rough surface. However, other inner surfaces of the elongated tube 114 can include smooth surfaces for an insect not to be able to walk on the sooth surfaces. In some examples, rough surfaces can be used to orient the insect to an imaging device and to move the insect to the second opening 112 or to the third cell 106. Other surfaces, for example, can be coated in non-stick polytetrafluoroethylene (PTFE, Fluon or Teflon) coatings to prevent the insect from adhering and walking on those surfaces. Thus, the imaging device in the imaging chamber 100 can capture an expected position or side of the insect. In some examples, the elongated tube 114 can include a scent, food, artificial light, natural light from the outside of the imaging chamber 100, or any other suitable means for an insect to pass through the elongated tube 114.--, in [0050], and, -- Yet further, traps may also have additional sensors such as weather sensors, daylight sensors, pollen sensors, dust sensors, air quality analyzers, GPS, or other sensors to collect and record additional data as described herein. And, other embodiments of a trap may have insect-specific adaptations, such as various colors inside the chamber (for contrast with the colors of various insects), various surfaces (e.g., sticky, rough, smooth, slanted, etc.), and various attractants (e.g., scents, CO.sub.2, UV light, etc) that can be alternatingly used.--, in [0094]; and also see:
--[0096] When motion has been detected, the insect trap or other local device will begin acquiring data from the portion of the chamber in which the motion was detected. This may include a continuous video acquisition, bursts of still images, or periodic video/image acquisition, or combinations thereof. As long as motion is still being detected and battery life permits, traps may continue to acquire insect data. (As described elsewhere herein, the insect data may include optical images, video clips, IR images, or images in which the insects are alternately exposed to other types of light such as UV, etc.)…. [0098] At step 608, the insect data (images, videos, etc.) can be preprocessed as described above with respect to FIG. 4, or via other computer vision techniques such as edge detection, segmentation (e.g., segmenting IR or UV images, or images in which an insect is dark against a light background, etc.) or the like. The full insect data is thus segmented into smaller portions of the data most likely to contain an insect.--, in [0096]-[0098];
Further see: --[0090] Once the deep learning model has been deployed and is being used to classify insects in images from chambers, traps or other such areas, a further tuning step can be employed to improve accuracy or otherwise modify the model. For example, in step 510, the model may optionally receive additional insect data, such as additional video clips, single images, or sets of images (which each may be solely optical, infrared, depth, etc.). This data may include actual images captures by traps/chambers associated with the system using the deep learning model, or may be acquired by other means (e.g., from individual photos or labs) from any locality or a locality associated with traps/chambers whose images are being classified by the model.--, in [0090];
--[0092] Additional information may include other image modalities or image conditions (e.g., strobing a UV light so that some images are acquired during UV exposure, or infrared images, zoom, etc.), or additional information that can be derived from the initially acquired image data. For example, the speed or method of movement of an insect could be determined in a variety of ways, including monitoring location of the insect within the trap frame by frame, and determining whether the insect is able to climb or attach to different surfaces within the trap.--, in [0092];
3.2 and, regarding to amended claim limitation “(iv) one or more infrared lights configured to illuminate the capture zone during image capture”
Dick (US 20190166823 A1) discloses one or more infrared lights configured to illuminate the capture zone during image capture (see DICK: e.g., -- the classifier training is accomplished by placing a digital camera in proximity to the baited trap to acquire training images. The trap doors are locked open, allowing animals to enter and exit unhindered. The camera contains a conventional complementary metal-oxide-semiconductor (CMOS) sensor with visible light and infrared wavelength detection. The camera also has infrared illumination that is autonomously turned on by the camera in low visible light conditions…. Preferably, this training image set includes at least 100 different images capturing different perspectives of each animal. Optionally, the training image set may be further expanded by converting some color images to greyscale to simulate infrared illumination images. In addition, an image augmentation procedure may be used to create random shifts, shears, rotations and flips to the images to create more image variations for each animal. A human operator defines the ground truth ROIs and animal classifications for each image used in the training set using an object tagging software tool on the first computer. In addition to the training images containing animals (positives), a set of non-animal images (negatives) are included in the training set for background reference. Ten percent of the positive training images are randomly selected and removed from the training set to serve as a test image set.--, in [0033]);
Dick and Patch are combinable as they are in the same field of endeavor: insects/pest, or animal traps with image capturing and image based insects/pest, or animal monitoring, identification/recognition, and classification. Therefore it would have been obvious to one of ordinary skill in the art at the time the invention was made to further modify Patch’s trap device using Dick’s teachings by including one or more infrared lights configured to illuminate the capture zone during image capture to Patch’s {the insect data may include optical images, video clips, IR images, or images in which the insects are alternately exposed to other types of light such as UV, etc.) in order to monitor/detect insects/pests in low visible light conditions (see Dick: e.g., at [0033]);
3.3 and, with regarding the amended limitation “ (iii) process the instance of image data via a classifier module, thereby to define detection data representative of identified presence of one or more insects of known insect types, wherein the classifier module is a trained classifier module trained using labelled images of insects of one or more insect types captured under infrared illumination”, which is still rejected under Patch as modified by DICK, and further in view of YU (CN 114332621 A), because:
Patch as modified by DICK further disclose (iii) process the instance of image data via a classifier module, thereby to define detection data representative of identified presence of one or more insects of known insect types, wherein the classifier module is a trained classifier module trained using labelled images of insects of one or more insect types captured under infrared illumination (see Patch: e.g., --[0043] Object Classification: An example current classifier model can discriminate between four insect orders: Coleoptera, Diptera, Hymenoptera and Lepidoptera (beetles, flies, wasps/bees, butterflies/moths) at above 90% accuracy. The example classifier model is a VGG16 that has been pretrained on ImageNet. In some examples, the output layer can be modified to have 13 output classes, each denoting a taxonomic label (e.g. Formicidae) within a taxonomic level (e.g. family), and then fine-tuned the network on 82,000 images of local insects obtained from iNaturalist and GBIF. In further examples, Tensorflow and Keras can be used to train and classify images.--, in [0043]; and,
--the imaging chamber 100 can further include the power supply and voltage converter to power all imaging devices 122, 130, 142 and computer components. In even further examples, the imaging chamber 100 can further include a non-transitory computer readable medium (e.g., memory, solid-state hard drive, etc.) to store the visual images. In even further examples, the imaging chamber 100 can include a transceiver to transmit the visual images to a server and/or receive weather data to synchronize with the captured data. In even further examples, the imaging chamber 100 may include a processor with a memory to transmit data to another imaging chamber or any other suitable remote location. In some examples, the processor in the imaging chamber 100 can provide classification results to other imaging chambers with different deep learning models to identify insects. This can allow for easy remote monitoring of several imaging chambers deployed in close proximity and sending out unknown images for prompt human intervention. --, in [0063]; and, --[0085] At step 502, a scope of classification of insects may be determined. In some instances, this may include a user or company generating or tuning the process specifying the types of insects that are desired to be classified (e.g., only butterflies, only mosquitos, only given classes or orders of insects, etc.). Alternatively, or in combination, a user or company may specify a geography in which the process will be utilized, such as a country, state, county, ecosystem, etc., from which a set of likely/native insects may be obtained from publicly available sources or pre-defined lists. This information may be obtained from a user through a user interface, or may be predetermined for different models/categories of products to be sold. In embodiments in which this information is obtained from a user, the user interface may, for example, allow a user to select from among various classes and orders of insects, various geographies, or the like. This information will be used to determine the output layer of a deep learning model that will classify insects, as well as the scope of training information needed to train and/or tune the deep learning model for maximum accuracy and predictive power.
[0086] At step 504, training data may be received or obtained that will include the classes, species, and types of insects to be identified. In some embodiments, this may include publicly available insect data (e.g., from iNaturalist), or on insect data obtained using cameras of a trap system as described herein. In further examples, the training data can comprise all or a subset of publicly available insect data, taking into account the location of interest to detect insects (e.g., location of imaging chambers) and/or classes or orders that the user wants to classify. For example, in one test embodiment, the inventors scraped around 450,000 insect images taken in the northeast USA that represented the four major orders: Diptera, Hymenoptera, Coleoptera, and Lepidoptera. In some examples, the specific training data (e.g., images or videos) can be obtained regarding the location of interest for the insect detection. For example, when the system is deployed to detect insects in Pennsylvania, the system can reduce the amount of training data (e.g., privately generated insect data only with insects living in Pennsylvania) by limiting to specific classes of insects native to Pennsylvania. In some instances, it may be beneficial to also obtain non-insect training data (e.g., surfaces with shadows, fog, haze, smoke, vegetation, cobwebs, etc.) that may be utilized to help improve classification power. The training data may include label information such as the class, order, family, genus and species of the insect, and/or the sex or role (e.g., queen) of the insect (if visually different). And, the images used in the training data can include any perspectives and/or poses of insects shown.–in [0085]-[0086]; also see DICK: e.g., ---- the classifier training is accomplished by placing a digital camera in proximity to the baited trap to acquire training images. The trap doors are locked open, allowing animals to enter and exit unhindered. The camera contains a conventional complementary metal-oxide-semiconductor (CMOS) sensor with visible light and infrared wavelength detection. The camera also has infrared illumination that is autonomously turned on by the camera in low visible light conditions…. Preferably, this training image set includes at least 100 different images capturing different perspectives of each animal. Optionally, the training image set may be further expanded by converting some color images to greyscale to simulate infrared illumination images. In addition, an image augmentation procedure may be used to create random shifts, shears, rotations and flips to the images to create more image variations for each animal. A human operator defines the ground truth ROIs and animal classifications for each image used in the training set using an object tagging software tool on the first computer. In addition to the training images containing animals (positives), a set of non-animal images (negatives) are included in the training set for background reference. Ten percent of the positive training images are randomly selected and removed from the training set to serve as a test image set.--, in [0033]);
Patch however does not exilically disclose labeled training image data are infrared images;
Yu discloses labeled training image data {for classification} are infrared images (see YU: e.g., -- recognition and treating insect pests based on multi-model feature fusion, the invention through visible light image, infrared image and hyperspectral image model training sample,… the advantages of comprehensive each of the neural network module to obtain the optimal solution, improving the generalization ability of the insect recognition --, in abstract, and, -- according to the training image set training pre-set pest recognition model: dividing the training image set into a plurality of subset, wherein each group of the subset comprises a set number of visible light image, an infrared image and a high-spectral image; pre-setting a plurality of neural network spaced with difference of the module, the plurality of groups subset subset input to each of the said neural network module, --, in page 2 of English version of CN 114332621 A, as provided in the Office Action; and, -- obtaining the label image set, the label image set comprises a plurality of visible light image with preset classification label, an infrared image and a high spectrum image;
inputting the label image set to the insect recognition obtaining a plurality of first features output by each of the said neural network module, and splicing a plurality of first features of each of the said neural network module, obtaining the second characteristic of each of the said neural network--, in page 3 of English version of CN 114332621 A, as provided in the Office Action);
YU and Patch (as modified by DICK) are combinable as they are in the same field of endeavor: image-based insects/pest, or animal monitoring, identification/recognition, and classification. Therefore it would have been obvious to one of ordinary skill in the art at the time the invention was made to further modify Patch (as modified by DICK)’s trap device using YU’s teachings by including labeled training infrared image data {for classification} to Patch (as modified by DICK)’s {the insect data may include optical images, video clips, IR images, or images in which the insects are alternately exposed to other types of light such as UV, etc.) in order to perform insect recognition (see YU: e.g., in abstract, and in pages 2-3 of English version of CN 114332621 A, as provided in the Office Action).
Therefore, claims 1, 6-19 are still not patentably distinguishable over the prior art reference(s). Further discussions are addressed in the prior art rejection section below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1, 6-19 are rejected under 35 U.S.C. 103(a) as being unpatentable over Patch (US 20220361471 A1, Date Filed: 2022-05-11), in view of Dick (US 20190166823 A1), and further in view of YU (CN 114332621 A).
Re Claim 1, Patch discloses a trap device configured to facilitate identification and capture of insects (see Patch: e. g., Fig. 1, --An intelligent insect trap and identification system is disclosed. The intelligent insect trap and identification system can include an insect imaging chamber and identification system.--, in abstract), the device including:
a trap assembly, the trap assembly (see Patch: e. g., Fig. 1, --The intelligent insect trap and identification system can include an insect imaging chamber and identification system. The chamber can include a first cell for accepting insects, a second cell, a first reflector in the second cell, and a first imaging device in the second cell for recording one or more first visual images of the one or more insects in the first cell. Based on the image, the insect imaging chamber can detect and identify the insects.--, in abstract) including:
(i) one or more passageways through which an insect is able to crawl, each of the one or more passageways aperture having an external opening and in internal opening (see Patch: e. g., Fig. 1, -- the insect guide tube 108 can include two openings 110, 112, an elongated tube 114, and/or a trapdoor 116. In some scenarios, a first opening 110 of the insect guide tube 108 can be close to the chamber opening 106….. a second opening 112 of the insect guide tube 108 can be an exit or and entrance of an insect. In some examples, a clear lid 118 can be kept on top of the second opening 112 of the imaging insect guide tube 108 to incentive an insect to leave the imaging chamber 100--; in [0050]-[0052]);
(ii) a cavity into which the internal openings feed (see Patch: e. g., Fig. 1,
-- the insect guide tube 108 can include two openings 110, 112, an elongated tube 114, and/or a trapdoor 116. In some scenarios, a first opening 110 of the insect guide tube 108 can be close to the chamber opening 106….. a second opening 112 of the insect guide tube 108 can be an exit or and entrance of an insect. In some examples, a clear lid 118 can be kept on top of the second opening 112 of the imaging insect guide tube 108 to incentive an insect to leave the imaging chamber 100--; in [0050]-[0052]); and
(iii) a pitfall trap arrangement within the cavity, such that an insect that crawls through one of the passageways and egresses through the internal opening of that passageway is transported into a capture zone of the pitfall trap arrangement, wherein the capture zone includes a capture surface on which the insect is maintained following transportation into the capture zone, the capture surface being configured to absorb infrared light, and wherein the capture surface is formed of a textured material configured to absorb infrared light (see Patch: e. g., Fig. 1, --example insect traps or imaging chambers are disclosed. An insect imaging chamber includes: a first cell for accepting one or more insects; a second cell, the second cell being separated from the first cell; a first reflector in the second cell;--, and, -- The example insect trap is time-efficient, non-lethal, and user-friendly. In addition, the example insect imaging chamber in the present disclosure can be non-lethal traps for insect biodiversity monitoring. The example insect trap and identification system can preprocess collected images (video frames, films, photographs, etc.) to reduce the number of images and only use part of the image (bounding boxes) for insect trap and identification. Accordingly, the example insect trap and identification system can effectively and efficiently detect and classify insects using an AI system.--, in [0004], and [0017];
-- the insect guide tube 108 can include two openings 110, 112, an elongated tube 114, and/or a trapdoor 116. In some scenarios, a first opening 110 of the insect guide tube 108 can be close to the chamber opening 106….. a second opening 112 of the insect guide tube 108 can be an exit or and entrance of an insect. In some examples, a clear lid 118 can be kept on top of the second opening 112 of the imaging insect guide tube 108 to incentive an insect to leave the imaging chamber 100--; in [0050]-[0052]; and, -- Yet further, traps may also have additional sensors such as weather sensors, daylight sensors, pollen sensors, dust sensors, air quality analyzers, GPS, or other sensors to collect and record additional data as described herein. And, other embodiments of a trap may have insect-specific adaptations, such as various colors inside the chamber (for contrast with the colors of various insects), various surfaces (e.g., sticky, rough, smooth, slanted, etc.), and various attractants (e.g., scents, CO.sub.2, UV light, etc) that can be alternatingly used.--, in [0094]; and also see:
--[0096] When motion has been detected, the insect trap or other local device will begin acquiring data from the portion of the chamber in which the motion was detected. This may include a continuous video acquisition, bursts of still images, or periodic video/image acquisition, or combinations thereof. As long as motion is still being detected and battery life permits, traps may continue to acquire insect data. (As described elsewhere herein, the insect data may include optical images, video clips, IR images, or images in which the insects are alternately exposed to other types of light such as UV, etc.)…. [0098] At step 608, the insect data (images, videos, etc.) can be preprocessed as described above with respect to FIG. 4, or via other computer vision techniques such as edge detection, segmentation (e.g., segmenting IR or UV images, or images in which an insect is dark against a light background, etc.) or the like. The full insect data is thus segmented into smaller portions of the data most likely to contain an insect.--, in [0096]-[0098];
Further see: --[0090] Once the deep learning model has been deployed and is being used to classify insects in images from chambers, traps or other such areas, a further tuning step can be employed to improve accuracy or otherwise modify the model. For example, in step 510, the model may optionally receive additional insect data, such as additional video clips, single images, or sets of images (which each may be solely optical, infrared, depth, etc.). This data may include actual images captures by traps/chambers associated with the system using the deep learning model, or may be acquired by other means (e.g., from individual photos or labs) from any locality or a locality associated with traps/chambers whose images are being classified by the model.--, in [0090];
--[0092] Additional information may include other image modalities or image conditions (e.g., strobing a UV light so that some images are acquired during UV exposure, or infrared images, zoom, etc.), or additional information that can be derived from the initially acquired image data. For example, the speed or method of movement of an insect could be determined in a variety of ways, including monitoring location of the insect within the trap frame by frame, and determining whether the insect is able to climb or attach to different surfaces within the trap.--, in [0092]);
a monitoring unit, the monitoring unit including:
(i) an image capture module configured to capture image data for a field of view that includes the capture zone (see Patch: e. g., Fig. 1, --[0017] Example intelligent insect trap and identification system (e.g., example insect imaging chambers and systems to process insect images) in the present disclosure can use a sophisticated camera system paired with novel software for insect detection and identification. The example insect trap is time-efficient, non-lethal, and user-friendly. In addition, the example insect imaging chamber in the present disclosure can be non-lethal traps for insect biodiversity monitoring. The example insect trap and identification system can preprocess collected images (video frames, films, photographs, etc.) to reduce the number of images and only use part of the image (bounding boxes) for insect trap and identification. Accordingly, the example insect trap and identification system can effectively and efficiently detect and classify insects using an AI system.--, in [0017]; and, -- [0034] The example insect trap and identification system can include two main components: an example insect imaging chamber and an example insect identification application. The insect trap and identification system can collect data. The example insect identification application can extract meaningful information from the data.
[0035] Hardware Stack: The example insect imaging chamber can represent a minimal baseline for collection of environmental and video data. The example insect imaging chamber can include a tube with two imaging devices (e.g., 12.3 megapixel HD cameras or any other suitable cameras) positioned such that they will capture the top and side views of insects as they crawl through the trap.--, in [0034]-[0035]);
(ii) a communications module (see Patch: e. g., Fig. 1, -- In some embodiments, in the second cell 104 one or more visual images (e.g., photograph, film, video, etc.) are recorded, stored, and/or transmitted via an imaging device and/or a controller. For example, the second cell 104 can include a first reflector 120 and a first imaging device 122 for recording one or more first visual images (e.g., photograph, film, video, etc.) of the insect…. the first imaging device 122 can be a digital camera, a video recording device, a camcorder, a motion picture camera; or any other suitable device capable of recording, storing, or transmitting visual images (e.g., photographs or videos) of insects.--, in [0055]-[0057], and [0063]; and, -- the imaging chamber 100 including one or more imaging devices can record the multiple images of the one or more objects and store the multiple images in a non-transitory computer-readable medium. In a non-limiting examples, the system can receive the multiple images stored in the non-transitory computer-readable medium via any suitable communication network or combination of communication networks (e.g., a Wi-Fi network, a peer-to-peer network, a cellular network, a wired network, etc.).--, in [0067]); and,
(iii) a processing unit that is configured to execute logical instructions thereby to cause the image capture module to capture images in accordance with a predefined capture protocol, and the communications module to communicate resultant image data to a remote processing system in accordance with a predefined transmission protocol (see Patch: e. g., Fig. 1, -- In some embodiments, in the second cell 104 one or more visual images (e.g., photograph, film, video, etc.) are recorded, stored, and/or transmitted via an imaging device and/or a controller. For example, the second cell 104 can include a first reflector 120 and a first imaging device 122 for recording one or more first visual images (e.g., photograph, film, video, etc.) of the insect…. the first imaging device 122 can be a digital camera, a video recording device, a camcorder, a motion picture camera; or any other suitable device capable of recording, storing, or transmitting visual images (e.g., photographs or videos) of insects.--, in [0055]-[0057], and, --the imaging chamber 100 can further include the power supply and voltage converter to power all imaging devices 122, 130, 142 and computer components. In even further examples, the imaging chamber 100 can further include a non-transitory computer readable medium (e.g., memory, solid-state hard drive, etc.) to store the visual images. In even further examples, the imaging chamber 100 can include a transceiver to transmit the visual images to a server and/or receive weather data to synchronize with the captured data. In even further examples, the imaging chamber 100 may include a processor with a memory to transmit data to another imaging chamber or any other suitable remote location. In some examples, the processor in the imaging chamber 100 can provide classification results to other imaging chambers with different deep learning models to identify insects. This can allow for easy remote monitoring of several imaging chambers deployed in close proximity and sending out unknown images for prompt human intervention. --, in [0063]; and, -- the imaging chamber 100 including one or more imaging devices can record the multiple images of the one or more objects and store the multiple images in a non-transitory computer-readable medium. In a non-limiting examples, the system can receive the multiple images stored in the non-transitory computer-readable medium via any suitable communication network or combination of communication networks (e.g., a Wi-Fi network, a peer-to-peer network, a cellular network, a wired network, etc.).--, in [0067]);
Patch however does not explicitly disclose (iv) one or more infrared lights configured to illuminate the capture zone during image capture”
Dick (US 20190166823 A1) discloses (iv) one or more infrared lights configured to illuminate the capture zone during image capture (see DICK: e.g., -- the classifier training is accomplished by placing a digital camera in proximity to the baited trap to acquire training images. The trap doors are locked open, allowing animals to enter and exit unhindered. The camera contains a conventional complementary metal-oxide-semiconductor (CMOS) sensor with visible light and infrared wavelength detection. The camera also has infrared illumination that is autonomously turned on by the camera in low visible light conditions…. Preferably, this training image set includes at least 100 different images capturing different perspectives of each animal. Optionally, the training image set may be further expanded by converting some color images to greyscale to simulate infrared illumination images. In addition, an image augmentation procedure may be used to create random shifts, shears, rotations and flips to the images to create more image variations for each animal. A human operator defines the ground truth ROIs and animal classifications for each image used in the training set using an object tagging software tool on the first computer. In addition to the training images containing animals (positives), a set of non-animal images (negatives) are included in the training set for background reference. Ten percent of the positive training images are randomly selected and removed from the training set to serve as a test image set.--, in [0033]);
Dick and Patch are combinable as they are in the same field of endeavor: insects/pest, or animal traps with image capturing and image based insects/pest, or animal monitoring, identification/recognition, and classification. Therefore it would have been obvious to one of ordinary skill in the art at the time the invention was made to further modify Patch’s trap device using Dick’s teachings by including (iv) one or more infrared lights configured to illuminate the capture zone during image capture to Patch’s {the insect data may include optical images, video clips, IR images, or images in which the insects are alternately exposed to other types of light such as UV, etc.) in order to monitor/detect insects/pests in low visible light conditions (see Dick: e.g., at [0033]);
Patch as modified by DICK further disclose wherein the remote processing system is configured to:
(i) receive a data transmission including at least one instance of image data transmitted by the trap device processing unit (see Patch: e. g., Fig. 1, -- In some embodiments, in the second cell 104 one or more visual images (e.g., photograph, film, video, etc.) are recorded, stored, and/or transmitted via an imaging device and/or a controller. For example, the second cell 104 can include a first reflector 120 and a first imaging device 122 for recording one or more first visual images (e.g., photograph, film, video, etc.) of the insect…. the first imaging device 122 can be a digital camera, a video recording device, a camcorder, a motion picture camera; or any other suitable device capable of recording, storing, or transmitting visual images (e.g., photographs or videos) of insects.--, in [0055]-[0057], and, --the imaging chamber 100 can further include the power supply and voltage converter to power all imaging devices 122, 130, 142 and computer components. In even further examples, the imaging chamber 100 can further include a non-transitory computer readable medium (e.g., memory, solid-state hard drive, etc.) to store the visual images. In even further examples, the imaging chamber 100 can include a transceiver to transmit the visual images to a server and/or receive weather data to synchronize with the captured data. In even further examples, the imaging chamber 100 may include a processor with a memory to transmit data to another imaging chamber or any other suitable remote location. In some examples, the processor in the imaging chamber 100 can provide classification results to other imaging chambers with different deep learning models to identify insects. This can allow for easy remote monitoring of several imaging chambers deployed in close proximity and sending out unknown images for prompt human intervention. --, in [0063]; and, -- the imaging chamber 100 including one or more imaging devices can record the multiple images of the one or more objects and store the multiple images in a non-transitory computer-readable medium. In a non-limiting examples, the system can receive the multiple images stored in the non-transitory computer-readable medium via any suitable communication network or combination of communication networks (e.g., a Wi-Fi network, a peer-to-peer network, a cellular network, a wired network, etc.).--, in [0067]);
(ii) determine a unique identifier representative of the processing unit (see Patch: e. g., Fig. 1, -- In some embodiments, in the second cell 104 one or more visual images (e.g., photograph, film, video, etc.) are recorded, stored, and/or transmitted via an imaging device and/or a controller. For example, the second cell 104 can include a first reflector 120 and a first imaging device 122 for recording one or more first visual images (e.g., photograph, film, video, etc.) of the insect…. the first imaging device 122 can be a digital camera, a video recording device, a camcorder, a motion picture camera; or any other suitable device capable of recording, storing, or transmitting visual images (e.g., photographs or videos) of insects.--, in [0055]-[0057], and, --the imaging chamber 100 can further include the power supply and voltage converter to power all imaging devices 122, 130, 142 and computer components. In even further examples, the imaging chamber 100 can further include a non-transitory computer readable medium (e.g., memory, solid-state hard drive, etc.) to store the visual images. In even further examples, the imaging chamber 100 can include a transceiver to transmit the visual images to a server and/or receive weather data to synchronize with the captured data. In even further examples, the imaging chamber 100 may include a processor with a memory to transmit data to another imaging chamber or any other suitable remote location. In some examples, the processor in the imaging chamber 100 can provide classification results to other imaging chambers with different deep learning models to identify insects. This can allow for easy remote monitoring of several imaging chambers deployed in close proximity and sending out unknown images for prompt human intervention. --, in [0063]; and, -- the imaging chamber 100 including one or more imaging devices can record the multiple images of the one or more objects and store the multiple images in a non-transitory computer-readable medium. In a non-limiting examples, the system can receive the multiple images stored in the non-transitory computer-readable medium via any suitable communication network or combination of communication networks (e.g., a Wi-Fi network, a peer-to-peer network, a cellular network, a wired network, etc.).--, in [0067]);
(iii) process the instance of image data via a classifier module, thereby to define detection data representative of identified presence of one or more insects of known insect types wherein the classifier module is a trained classifier module trained using labelled images of insects of one or more insect types captured under infrared illumination (see Patch: e.g., --[0043] Object Classification: An example current classifier model can discriminate between four insect orders: Coleoptera, Diptera, Hymenoptera and Lepidoptera (beetles, flies, wasps/bees, butterflies/moths) at above 90% accuracy. The example classifier model is a VGG16 that has been pretrained on ImageNet. In some examples, the output layer can be modified to have 13 output classes, each denoting a taxonomic label (e.g. Formicidae) within a taxonomic level (e.g. family), and then fine-tuned the network on 82,000 images of local insects obtained from iNaturalist and GBIF. In further examples, Tensorflow and Keras can be used to train and classify images.--, in [0043]; and,
--the imaging chamber 100 can further include the power supply and voltage converter to power all imaging devices 122, 130, 142 and computer components. In even further examples, the imaging chamber 100 can further include a non-transitory computer readable medium (e.g., memory, solid-state hard drive, etc.) to store the visual images. In even further examples, the imaging chamber 100 can include a transceiver to transmit the visual images to a server and/or receive weather data to synchronize with the captured data. In even further examples, the imaging chamber 100 may include a processor with a memory to transmit data to another imaging chamber or any other suitable remote location. In some examples, the processor in the imaging chamber 100 can provide classification results to other imaging chambers with different deep learning models to identify insects. This can allow for easy remote monitoring of several imaging chambers deployed in close proximity and sending out unknown images for prompt human intervention. --, in [0063]; and, --[0085] At step 502, a scope of classification of insects may be determined. In some instances, this may include a user or company generating or tuning the process specifying the types of insects that are desired to be classified (e.g., only butterflies, only mosquitos, only given classes or orders of insects, etc.). Alternatively, or in combination, a user or company may specify a geography in which the process will be utilized, such as a country, state, county, ecosystem, etc., from which a set of likely/native insects may be obtained from publicly available sources or pre-defined lists. This information may be obtained from a user through a user interface, or may be predetermined for different models/categories of products to be sold. In embodiments in which this information is obtained from a user, the user interface may, for example, allow a user to select from among various classes and orders of insects, various geographies, or the like. This information will be used to determine the output layer of a deep learning model that will classify insects, as well as the scope of training information needed to train and/or tune the deep learning model for maximum accuracy and predictive power.
[0086] At step 504, training data may be received or obtained that will include the classes, species, and types of insects to be identified. In some embodiments, this may include publicly available insect data (e.g., from iNaturalist), or on insect data obtained using cameras of a trap system as described herein. In further examples, the training data can comprise all or a subset of publicly available insect data, taking into account the location of interest to detect insects (e.g., location of imaging chambers) and/or classes or orders that the user wants to classify. For example, in one test embodiment, the inventors scraped around 450,000 insect images taken in the northeast USA that represented the four major orders: Diptera, Hymenoptera, Coleoptera, and Lepidoptera. In some examples, the specific training data (e.g., images or videos) can be obtained regarding the location of interest for the insect detection. For example, when the system is deployed to detect insects in Pennsylvania, the system can reduce the amount of training data (e.g., privately generated insect data only with insects living in Pennsylvania) by limiting to specific classes of insects native to Pennsylvania. In some instances, it may be beneficial to also obtain non-insect training data (e.g., surfaces with shadows, fog, haze, smoke, vegetation, cobwebs, etc.) that may be utilized to help improve classification power. The training data may include label information such as the class, order, family, genus and species of the insect, and/or the sex or role (e.g., queen) of the insect (if visually different). And, the images used in the training data can include any perspectives and/or poses of insects shown.–in [0085]-[0086]; also see: and, -- In some embodiments, in the second cell 104 one or more visual images (e.g., photograph, film, video, etc.) are recorded, stored, and/or transmitted via an imaging device and/or a controller. For example, the second cell 104 can include a first reflector 120 and a first imaging device 122 for recording one or more first visual images (e.g., photograph, film, video, etc.) of the insect…. the first imaging device 122 can be a digital camera, a video recording device, a camcorder, a motion picture camera; or any other suitable device capable of recording, storing, or transmitting visual images (e.g., photographs or videos) of insects.--, in [0055]-[0057]; -- the imaging chamber 100 including one or more imaging devices can record the multiple images of the one or more objects and store the multiple images in a non-transitory computer-readable medium. In a non-limiting examples, the system can receive the multiple images stored in the non-transitory computer-readable medium via any suitable communication network or combination of communication networks (e.g., a Wi-Fi network, a peer-to-peer network, a cellular network, a wired network, etc.).--, in [0067], and, --to perform a cyclical or self-learning approach at step 514 to providing additional classification power to the algorithm/process. In some embodiments, a regression or other statistical method may be performed to associate factors like time of day, temperature, time from/to sunrise/sunset, etc. with high confidence-score classifications of insects. Thus, a weighting factor can be associated with any of these factors. Then, when an image is analyzed by the deep learning model and an insufficiently-high confidence score (or two or more similar confidence scores) is returned for a given classification, the presence of additional factors at the time the image(s) were acquired can be considered. If, e.g., time of day, season, or temperature might highly correlate with a given species among the possible classifications, but none of the others, the algorithm can give a preliminary or tentative classification of the given species and notify a user of the factor that was used to supplement the prediction. Or, if an insect is able to climb a specific type of surface, which other likely insects are not, this information could rule out some results of the classifier model. In other embodiments, the additional/environmental data can be combined with image data and provided to the deep learning model as training data, such that a new or re-tuned model can be generated directly from the additional/environmental data. In a non-limiting scenario, the system can receive environmental information from the user, any suitable device (e.g., insect imaging chamber 100, etc.) and/or any suitable database (e.g., National Oceanic and Atmospheric Administration (NOAA) Online Weather Database, etc.).--, in [0093];
also see DICK: e.g., ---- the classifier training is accomplished by placing a digital camera in proximity to the baited trap to acquire training images. The trap doors are locked open, allowing animals to enter and exit unhindered. The camera contains a conventional complementary metal-oxide-semiconductor (CMOS) sensor with visible light and infrared wavelength detection. The camera also has infrared illumination that is autonomously turned on by the camera in low visible light conditions…. Preferably, this training image set includes at least 100 different images capturing different perspectives of each animal. Optionally, the training image set may be further expanded by converting some color images to greyscale to simulate infrared illumination images. In addition, an image augmentation procedure may be used to create random shifts, shears, rotations and flips to the images to create more image variations for each animal. A human operator defines the ground truth ROIs and animal classifications for each image used in the training set using an object tagging software tool on the first computer. In addition to the training images containing animals (positives), a set of non-animal images (negatives) are included in the training set for background reference. Ten percent of the positive training images are randomly selected and removed from the training set to serve as a test image set.--, in [0033]);
Patch however does not exilically disclose labeled training image data are infrared images;
Yu discloses labeled training image data {for classification} are infrared images (see YU: e.g., -- recognition and treating insect pests based on multi-model feature fusion, the invention through visible light image, infrared image and hyperspectral image model training sample,… the advantages of comprehensive each of the neural network module to obtain the optimal solution, improving the generalization ability of the insect recognition --, in abstract, and, -- according to the training image set training pre-set pest recognition model: dividing the training image set into a plurality of subset, wherein each group of the subset comprises a set number of visible light image, an infrared image and a high-spectral image; pre-setting a plurality of neural network spaced with difference of the module, the plurality of groups subset subset input to each of the said neural network module, --, in page 2 of English version of CN 114332621 A, as provided in the Office Action; and, -- obtaining the label image set, the label image set comprises a plurality of visible light image with preset classification label, an infrared image and a high spectrum image;
inputting the label image set to the insect recognition obtaining a plurality of first features output by each of the said neural network module, and splicing a plurality of first features of each of the said neural network module, obtaining the second characteristic of each of the said neural network--, in page 3 of English version of CN 114332621 A, as provided in the Office Action);
YU and Patch (as modified by DICK) are combinable as they are in the same field of endeavor: image-based insects/pest, or animal monitoring, identification/recognition, and classification. Therefore it would have been obvious to one of ordinary skill in the art at the time the invention was made to further modify Patch (as modified by DICK)’s trap device using YU’s teachings by including labeled training infrared image data {for classification} to Patch (as modified by DICK)’s {the insect data may include optical images, video clips, IR images, or images in which the insects are alternately exposed to other types of light such as UV, etc.) in order to perform insect recognition (see YU: e.g., in abstract, and in pages 2-3 of English version of CN 114332621 A, as provided in the Office Action); and,
Patch as modified by DICK and YU further disclose (iv) record the detection data in a database such that the detection data is associated with the processing unit via the unique identifier of the processing unit (see Patch: e. g., Fig. 1, --[0043] Object Classification: An example current classifier model can discriminate between four insect orders: Coleoptera, Diptera, Hymenoptera and Lepidoptera (beetles, flies, wasps/bees, butterflies/moths) at above 90% accuracy. The example classifier model is a VGG16 that has been pretrained on ImageNet. In some examples, the output layer can be modified to have 13 output classes, each denoting a taxonomic label (e.g. Formicidae) within a taxonomic level (e.g. family), and then fine-tuned the network on 82,000 images of local insects obtained from iNaturalist and GBIF. In further examples, Tensorflow and Keras can be used to train and classify images.--, in [0043]
-- In some embodiments, in the second cell 104 one or more visual images (e.g., photograph, film, video, etc.) are recorded, stored, and/or transmitted via an imaging device and/or a controller. For example, the second cell 104 can include a first reflector 120 and a first imaging device 122 for recording one or more first visual images (e.g., photograph, film, video, etc.) of the insect…. the first imaging device 122 can be a digital camera, a video recording device, a camcorder, a motion picture camera; or any other suitable device capable of recording, storing, or transmitting visual images (e.g., photographs or videos) of insects.--, in [0055]-[0057], and, --the imaging chamber 100 can further include the power supply and voltage converter to power all imaging devices 122, 130, 142 and computer components. In even further examples, the imaging chamber 100 can further include a non-transitory computer readable medium (e.g., memory, solid-state hard drive, etc.) to store the visual images. In even further examples, the imaging chamber 100 can include a transceiver to transmit the visual images to a server and/or receive weather data to synchronize with the captured data. In even further examples, the imaging chamber 100 may include a processor with a memory to transmit data to another imaging chamber or any other suitable remote location. In some examples, the processor in the imaging chamber 100 can provide classification results to other imaging chambers with different deep learning models to identify insects. This can allow for easy remote monitoring of several imaging chambers deployed in close proximity and sending out unknown images for prompt human intervention. --, in [0063]; and, -- the imaging chamber 100 including one or more imaging devices can record the multiple images of the one or more objects and store the multiple images in a non-transitory computer-readable medium. In a non-limiting examples, the system can receive the multiple images stored in the non-transitory computer-readable medium via any suitable communication network or combination of communication networks (e.g., a Wi-Fi network, a peer-to-peer network, a cellular network, a wired network, etc.).--, in [0067], and, --to perform a cyclical or self-learning approach at step 514 to providing additional classification power to the algorithm/process. In some embodiments, a regression or other statistical method may be performed to associate factors like time of day, temperature, time from/to sunrise/sunset, etc. with high confidence-score classifications of insects. Thus, a weighting factor can be associated with any of these factors. Then, when an image is analyzed by the deep learning model and an insufficiently-high confidence score (or two or more similar confidence scores) is returned for a given classification, the presence of additional factors at the time the image(s) were acquired can be considered. If, e.g., time of day, season, or temperature might highly correlate with a given species among the possible classifications, but none of the others, the algorithm can give a preliminary or tentative classification of the given species and notify a user of the factor that was used to supplement the prediction. Or, if an insect is able to climb a specific type of surface, which other likely insects are not, this information could rule out some results of the classifier model. In other embodiments, the additional/environmental data can be combined with image data and provided to the deep learning model as training data, such that a new or re-tuned model can be generated directly from the additional/environmental data. In a non-limiting scenario, the system can receive environmental information from the user, any suitable device (e.g., insect imaging chamber 100, etc.) and/or any suitable database (e.g., National Oceanic and Atmospheric Administration (NOAA) Online Weather Database, etc.).--, in [0093]).
Re Claim 6, as modified by DICK and YU further disclose wherein the capture surface is formed of a textured plastic (see Patch: e.g., -- the elongated tube 114 could be a cuboid shape. However, the elongated tube 114 could be any other suitable shape (e.g., a cubic shape, a cylindrical shape, etc.). In a non-limiting example, the size of the insect entrance of the elongated tube 114 can be designed for an insect to move in a limited and predicted way. The insect entrance of the elongated tube 114 can be a part of the elongated tube 114, which is connected to the first opening 110. An imaging device can record one or more visual images (e.g., photograph, film, video, or any other suitable image) of an expected position of the insect when the insect moves through the elongated tube 114. It should be appreciated that the size of the insect entrance of the elongated tube 114 can be big enough to accept multiple insects at the same time. In another example, the size of the insect entrance of the elongated tube 114 can be designed based on the sizes of insects that the user wants to analyze. Movement and positioning of an insect can also be influenced by alternative surface composition within the insect guide tube 108. For example, one inner surface of the elongated tube 114 can include a rough or adhesive surface for an insect to walk on the rough surface. However, other inner surfaces of the elongated tube 114 can include smooth surfaces for an insect not to be able to walk on the sooth surfaces. In some examples, rough surfaces can be used to orient the insect to an imaging device and to move the insect to the second opening 112 or to the third cell 106. Other surfaces, for example, can be coated in non-stick polytetrafluoroethylene (PTFE, Fluon or Teflon) coatings to prevent the insect from adhering and walking on those surfaces. Thus, the imaging device in the imaging chamber 100 can capture an expected position or side of the insect. In some examples, the elongated tube 114 can include a scent, food, artificial light, natural light from the outside of the imaging chamber 100, or any other suitable means for an insect to pass through the elongated tube 114.--, in [0050], and, -- Yet further, traps may also have additional sensors such as weather sensors, daylight sensors, pollen sensors, dust sensors, air quality analyzers, GPS, or other sensors to collect and record additional data as described herein. And, other embodiments of a trap may have insect-specific adaptations, such as various colors inside the chamber (for contrast with the colors of various insects), various surfaces (e.g., sticky, rough, smooth, slanted, etc.), and various attractants (e.g., scents, CO.sub.2, UV light, etc) that can be alternatingly used.--, in [0094]).
Re Claim 7, Patch as modified by DICK and YU further disclose wherein the trap assembly includes a base and a sidewall assembly upwardly extending from the base to a sidewall assembly top edge, wherein the one or more passageways are formed though the sidewall assembly (see Patch: e. g., Fig. 1, -- the insect guide tube 108 can include two openings 110, 112, an elongated tube 114, and/or a trapdoor 116. In some scenarios, a first opening 110 of the insect guide tube 108 can be close to the chamber opening 106….. a second opening 112 of the insect guide tube 108 can be an exit or and entrance of an insect. In some examples, a clear lid 118 can be kept on top of the second opening 112 of the imaging insect guide tube 108 to incentive an insect to leave the imaging chamber 100--; in [0050]-[0052]).
Re Claim 8, Patch as modified by DICK and YU further disclose wherein the one or more passageways are formed by gaps between the sidewall assembly top edge and a base of the monitoring unit (see Patch: e. g., Fig. 1, -- In some embodiments, in the second cell 104 one or more visual images (e.g., photograph, film, video, etc.) are recorded, stored, and/or transmitted via an imaging device and/or a controller. For example, the second cell 104 can include a first reflector 120 and a first imaging device 122 for recording one or more first visual images (e.g., photograph, film, video, etc.) of the insect…. the first imaging device 122 can be a digital camera, a video recording device, a camcorder, a motion picture camera; or any other suitable device capable of recording, storing, or transmitting visual images (e.g., photographs or videos) of insects.--, in [0043]-[0045], [0050]-[0052], and [0055]-[0057], and, --the imaging chamber 100 can further include the power supply and voltage converter to power all imaging devices 122, 130, 142 and computer components. In even further examples, the imaging chamber 100 can further include a non-transitory computer readable medium (e.g., memory, solid-state hard drive, etc.) to store the visual images. In even further examples, the imaging chamber 100 can include a transceiver to transmit the visual images to a server and/or receive weather data to synchronize with the captured data. In even further examples, the imaging chamber 100 may include a processor with a memory to transmit data to another imaging chamber or any other suitable remote location. In some examples, the processor in the imaging chamber 100 can provide classification results to other imaging chambers with different deep learning models to identify insects. This can allow for easy remote monitoring of several imaging chambers deployed in close proximity and sending out unknown images for prompt human intervention. --, in [0063]; and, -- the imaging chamber 100 including one or more imaging devices can record the multiple images of the one or more objects and store the multiple images in a non-transitory computer-readable medium. In a non-limiting examples, the system can receive the multiple images stored in the non-transitory computer-readable medium via any suitable communication network or combination of communication networks (e.g., a Wi-Fi network, a peer-to-peer network, a cellular network, a wired network, etc.).--, in [0067]).
Re Claim 9, Patch as modified by DICK and YU further disclose wherein the monitoring unit is housed in a monitoring unit body of the trap assembly, the monitoring unit body mounted to the sidewall assembly top edge via one or more connector members, wherein the one or more connector members define the openings such that each opening is bound at vertical sides thereof by edges of adjacent connector members, and at horizontal sides thereof by the sidewall assembly and the monitoring unit (see Patch: e. g., Fig. 1, -- In some embodiments, in the second cell 104 one or more visual images (e.g., photograph, film, video, etc.) are recorded, stored, and/or transmitted via an imaging device and/or a controller. For example, the second cell 104 can include a first reflector 120 and a first imaging device 122 for recording one or more first visual images (e.g., photograph, film, video, etc.) of the insect…. the first imaging device 122 can be a digital camera, a video recording device, a camcorder, a motion picture camera; or any other suitable device capable of recording, storing, or transmitting visual images (e.g., photographs or videos) of insects.--, in [0043]-[0045], [0050]-[0052], and [0055]-[0057], and, --the imaging chamber 100 can further include the power supply and voltage converter to power all imaging devices 122, 130, 142 and computer components. In even further examples, the imaging chamber 100 can further include a non-transitory computer readable medium (e.g., memory, solid-state hard drive, etc.) to store the visual images. In even further examples, the imaging chamber 100 can include a transceiver to transmit the visual images to a server and/or receive weather data to synchronize with the captured data. In even further examples, the imaging chamber 100 may include a processor with a memory to transmit data to another imaging chamber or any other suitable remote location. In some examples, the processor in the imaging chamber 100 can provide classification results to other imaging chambers with different deep learning models to identify insects. This can allow for easy remote monitoring of several imaging chambers deployed in close proximity and sending out unknown images for prompt human intervention. --, in [0063]; and, -- the imaging chamber 100 including one or more imaging devices can record the multiple images of the one or more objects and store the multiple images in a non-transitory computer-readable medium. In a non-limiting examples, the system can receive the multiple images stored in the non-transitory computer-readable medium via any suitable communication network or combination of communication networks (e.g., a Wi-Fi network, a peer-to-peer network, a cellular network, a wired network, etc.).--, in [0067]).
Re Claim 10, Patch as modified by DICK and YU further disclose wherein the monitoring unit body covers a top of the cavity (see Patch: e. g., Fig. 1, Fig.4, --[0017] Example intelligent insect trap and identification system (e.g., example insect imaging chambers and systems to process insect images) in the present disclosure can use a sophisticated camera system paired with novel software for insect detection and identification. The example insect trap is time-efficient, non-lethal, and user-friendly. In addition, the example insect imaging chamber in the present disclosure can be non-lethal traps for insect biodiversity monitoring. The example insect trap and identification system can preprocess collected images (video frames, films, photographs, etc.) to reduce the number of images and only use part of the image (bounding boxes) for insect trap and identification. Accordingly, the example insect trap and identification system can effectively and efficiently detect and classify insects using an AI system.--, in [0017]; and, -- [0034] The example insect trap and identification system can include two main components: an example insect imaging chamber and an example insect identification application. The insect trap and identification system can collect data. The example insect identification application can extract meaningful information from the data.
[0035] Hardware Stack: The example insect imaging chamber can represent a minimal baseline for collection of environmental and video data. The example insect imaging chamber can include a tube with two imaging devices (e.g., 12.3 megapixel HD cameras or any other suitable cameras) positioned such that they will capture the top and side views of insects as they crawl through the trap.--, in [0034]-[0035], and, --The second side image of the one or more insects can be different than the first side image captured from the first imaging device 122. For example, the first imaging device 122 can capture a top-view image of the insect based on the first reflector 120 while the second imaging device 130 can capture a side-view image of the insect based on the second reflector 128.--, in [0058]) .
Re Claim 11, Patch as modified by DICK and YU further disclose wherein the monitoring unit body includes a base surface configured to enable: (i) downward image capture by the image capture module through the cavity toward the capture zone (see Patch: e. g., Fig. 1, --[0043] Object Classification: An example current classifier model can discriminate between four insect orders: Coleoptera, Diptera, Hymenoptera and Lepidoptera (beetles, flies, wasps/bees, butterflies/moths) at above 90% accuracy. The example classifier model is a VGG16 that has been pretrained on ImageNet. In some examples, the output layer can be modified to have 13 output classes, each denoting a taxonomic label (e.g. Formicidae) within a taxonomic level (e.g. family), and then fine-tuned the network on 82,000 images of local insects obtained from iNaturalist and GBIF. In further examples, Tensorflow and Keras can be used to train and classify images.--, in [0043]
-- In some embodiments, in the second cell 104 one or more visual images (e.g., photograph, film, video, etc.) are recorded, stored, and/or transmitted via an imaging device and/or a controller. For example, the second cell 104 can include a first reflector 120 and a first imaging device 122 for recording one or more first visual images (e.g., photograph, film, video, etc.) of the insect…. the first imaging device 122 can be a digital camera, a video recording device, a camcorder, a motion picture camera; or any other suitable device capable of recording, storing, or transmitting visual images (e.g., photographs or videos) of insects.--, in [0055]-[0057], and, --the imaging chamber 100 can further include the power supply and voltage converter to power all imaging devices 122, 130, 142 and computer components. In even further examples, the imaging chamber 100 can further include a non-transitory computer readable medium (e.g., memory, solid-state hard drive, etc.) to store the visual images. In even further examples, the imaging chamber 100 can include a transceiver to transmit the visual images to a server and/or receive weather data to synchronize with the captured data. In even further examples, the imaging chamber 100 may include a processor with a memory to transmit data to another imaging chamber or any other suitable remote location. In some examples, the processor in the imaging chamber 100 can provide classification results to other imaging chambers with different deep learning models to identify insects. This can allow for easy remote monitoring of several imaging chambers deployed in close proximity and sending out unknown images for prompt human intervention. --, in [0063]; and, -- the imaging chamber 100 including one or more imaging devices can record the multiple images of the one or more objects and store the multiple images in a non-transitory computer-readable medium. In a non-limiting examples, the system can receive the multiple images stored in the non-transitory computer-readable medium via any suitable communication network or combination of communication networks (e.g., a Wi-Fi network, a peer-to-peer network, a cellular network, a wired network, etc.).--, in [0067], and, --to perform a cyclical or self-learning approach at step 514 to providing additional classification power to the algorithm/process. In some embodiments, a regression or other statistical method may be performed to associate factors like time of day, temperature, time from/to sunrise/sunset, etc. with high confidence-score classifications of insects. Thus, a weighting factor can be associated with any of these factors. Then, when an image is analyzed by the deep learning model and an insufficiently-high confidence score (or two or more similar confidence scores) is returned for a given classification, the presence of additional factors at the time the image(s) were acquired can be considered. If, e.g., time of day, season, or temperature might highly correlate with a given species among the possible classifications, but none of the others, the algorithm can give a preliminary or tentative classification of the given species and notify a user of the factor that was used to supplement the prediction. Or, if an insect is able to climb a specific type of surface, which other likely insects are not, this information could rule out some results of the classifier model. In other embodiments, the additional/environmental data can be combined with image data and provided to the deep learning model as training data, such that a new or re-tuned model can be generated directly from the additional/environmental data. In a non-limiting scenario, the system can receive environmental information from the user, any suitable device (e.g., insect imaging chamber 100, etc.) and/or any suitable database (e.g., National Oceanic and Atmospheric Administration (NOAA) Online Weather Database, etc.).--, in [0093])
Re Claim 12, Patch as modified by DICK and YU further disclose wherein the sidewall assembly tapers inward between the base and the sidewall assembly top edge (see Patch: e. g., Fig. 1, -- In some embodiments, in the second cell 104 one or more visual images (e.g., photograph, film, video, etc.) are recorded, stored, and/or transmitted via an imaging device and/or a controller. For example, the second cell 104 can include a first reflector 120 and a first imaging device 122 for recording one or more first visual images (e.g., photograph, film, video, etc.) of the insect…. the first imaging device 122 can be a digital camera, a video recording device, a camcorder, a motion picture camera; or any other suitable device capable of recording, storing, or transmitting visual images (e.g., photographs or videos) of insects.--, in [0043]-[0045], [0050]-[0052], and [0055]-[0057], and, --the imaging chamber 100 can further include the power supply and voltage converter to power all imaging devices 122, 130, 142 and computer components. In even further examples, the imaging chamber 100 can further include a non-transitory computer readable medium (e.g., memory, solid-state hard drive, etc.) to store the visual images. In even further examples, the imaging chamber 100 can include a transceiver to transmit the visual images to a server and/or receive weather data to synchronize with the captured data. In even further examples, the imaging chamber 100 may include a processor with a memory to transmit data to another imaging chamber or any other suitable remote location. In some examples, the processor in the imaging chamber 100 can provide classification results to other imaging chambers with different deep learning models to identify insects. This can allow for easy remote monitoring of several imaging chambers deployed in close proximity and sending out unknown images for prompt human intervention. --, in [0063]; and, -- the imaging chamber 100 including one or more imaging devices can record the multiple images of the one or more objects and store the multiple images in a non-transitory computer-readable medium. In a non-limiting examples, the system can receive the multiple images stored in the non-transitory computer-readable medium via any suitable communication network or combination of communication networks (e.g., a Wi-Fi network, a peer-to-peer network, a cellular network, a wired network, etc.).--, in [0067]).
Re Claim 13, Patch as modified by DICK and YU further disclose wherein the cavity is encircled by a cavity sidewall, and wherein the cavity sidewall is configured to inhibit upward crawling by an insect (see Patch: e. g., Fig. 1, -- In some embodiments, in the second cell 104 one or more visual images (e.g., photograph, film, video, etc.) are recorded, stored, and/or transmitted via an imaging device and/or a controller. For example, the second cell 104 can include a first reflector 120 and a first imaging device 122 for recording one or more first visual images (e.g., photograph, film, video, etc.) of the insect…. the first imaging device 122 can be a digital camera, a video recording device, a camcorder, a motion picture camera; or any other suitable device capable of recording, storing, or transmitting visual images (e.g., photographs or videos) of insects.--, in [0043]-[0045], [0050]-[0052], and [0055]-[0057], and, --the imaging chamber 100 can further include the power supply and voltage converter to power all imaging devices 122, 130, 142 and computer components. In even further examples, the imaging chamber 100 can further include a non-transitory computer readable medium (e.g., memory, solid-state hard drive, etc.) to store the visual images. In even further examples, the imaging chamber 100 can include a transceiver to transmit the visual images to a server and/or receive weather data to synchronize with the captured data. In even further examples, the imaging chamber 100 may include a processor with a memory to transmit data to another imaging chamber or any other suitable remote location. In some examples, the processor in the imaging chamber 100 can provide classification results to other imaging chambers with different deep learning models to identify insects. This can allow for easy remote monitoring of several imaging chambers deployed in close proximity and sending out unknown images for prompt human intervention. --, in [0063]; and, -- the imaging chamber 100 including one or more imaging devices can record the multiple images of the one or more objects and store the multiple images in a non-transitory computer-readable medium. In a non-limiting examples, the system can receive the multiple images stored in the non-transitory computer-readable medium via any suitable communication network or combination of communication networks (e.g., a Wi-Fi network, a peer-to-peer network, a cellular network, a wired network, etc.).--, in [0067]).
Re Claim 14, Patch as modified by DICK and YU further disclose wherein the cavity sidewall is formed of a smooth material configured to inhibit upward crawling by an insect (see Patch: e. g., Fig. 1, Fig.4, --[0017] Example intelligent insect trap and identification system (e.g., example insect imaging chambers and systems to process insect images) in the present disclosure can use a sophisticated camera system paired with novel software for insect detection and identification. The example insect trap is time-efficient, non-lethal, and user-friendly. In addition, the example insect imaging chamber in the present disclosure can be non-lethal traps for insect biodiversity monitoring. The example insect trap and identification system can preprocess collected images (video frames, films, photographs, etc.) to reduce the number of images and only use part of the image (bounding boxes) for insect trap and identification. Accordingly, the example insect trap and identification system can effectively and efficiently detect and classify insects using an AI system.--, in [0017]; and, -- [0034] The example insect trap and identification system can include two main components: an example insect imaging chamber and an example insect identification application. The insect trap and identification system can collect data. The example insect identification application can extract meaningful information from the data.
[0035] Hardware Stack: The example insect imaging chamber can represent a minimal baseline for collection of environmental and video data. The example insect imaging chamber can include a tube with two imaging devices (e.g., 12.3 megapixel HD cameras or any other suitable cameras) positioned such that they will capture the top and side views of insects as they crawl through the trap.--, in [0034]-[0035], and, --The second side image of the one or more insects can be different than the first side image captured from the first imaging device 122. For example, the first imaging device 122 can capture a top-view image of the insect based on the first reflector 120 while the second imaging device 130 can capture a side-view image of the insect based on the second reflector 128.--, in [0058]).
Re Claim 15, Patch as modified by DICK and YU further disclose wherein the cavity sidewall tapers inwardly from a top edge to a bottom edge (see Patch: e. g., Fig. 1, Fig.4, --[0017] Example intelligent insect trap and identification system (e.g., example insect imaging chambers and systems to process insect images) in the present disclosure can use a sophisticated camera system paired with novel software for insect detection and identification. The example insect trap is time-efficient, non-lethal, and user-friendly. In addition, the example insect imaging chamber in the present disclosure can be non-lethal traps for insect biodiversity monitoring. The example insect trap and identification system can preprocess collected images (video frames, films, photographs, etc.) to reduce the number of images and only use part of the image (bounding boxes) for insect trap and identification. Accordingly, the example insect trap and identification system can effectively and efficiently detect and classify insects using an AI system.--, in [0017]; and, -- [0034] The example insect trap and identification system can include two main components: an example insect imaging chamber and an example insect identification application. The insect trap and identification system can collect data. The example insect identification application can extract meaningful information from the data.
[0035] Hardware Stack: The example insect imaging chamber can represent a minimal baseline for collection of environmental and video data. The example insect imaging chamber can include a tube with two imaging devices (e.g., 12.3 megapixel HD cameras or any other suitable cameras) positioned such that they will capture the top and side views of insects as they crawl through the trap.--, in [0034]-[0035], and, --The second side image of the one or more insects can be different than the first side image captured from the first imaging device 122. For example, the first imaging device 122 can capture a top-view image of the insect based on the first reflector 120 while the second imaging device 130 can capture a side-view image of the insect based on the second reflector 128.--, in [0058]).
Re Claim 16, Patch as modified by DICK and YU further disclose wherein the bottom edge adjoins the capture zone (see Patch: e. g., Fig. 1, Fig.4, --[0017] Example intelligent insect trap and identification system (e.g., example insect imaging chambers and systems to process insect images) in the present disclosure can use a sophisticated camera system paired with novel software for insect detection and identification. The example insect trap is time-efficient, non-lethal, and user-friendly. In addition, the example insect imaging chamber in the present disclosure can be non-lethal traps for insect biodiversity monitoring. The example insect trap and identification system can preprocess collected images (video frames, films, photographs, etc.) to reduce the number of images and only use part of the image (bounding boxes) for insect trap and identification. Accordingly, the example insect trap and identification system can effectively and efficiently detect and classify insects using an AI system.--, in [0017]; and, -- [0034] The example insect trap and identification system can include two main components: an example insect imaging chamber and an example insect identification application. The insect trap and identification system can collect data. The example insect identification application can extract meaningful information from the data.
[0035] Hardware Stack: The example insect imaging chamber can represent a minimal baseline for collection of environmental and video data. The example insect imaging chamber can include a tube with two imaging devices (e.g., 12.3 megapixel HD cameras or any other suitable cameras) positioned such that they will capture the top and side views of insects as they crawl through the trap.--, in [0034]-[0035], and, --The second side image of the one or more insects can be different than the first side image captured from the first imaging device 122. For example, the first imaging device 122 can capture a top-view image of the insect based on the first reflector 120 while the second imaging device 130 can capture a side-view image of the insect based on the second reflector 128.--, in [0058]).
Re Claim 17, Patch as modified by DICK and YU further disclose a removable lure module mounted proximate to the capture zone (see Patch: e. g., Fig. 1, Fig.4, --[0017] Example intelligent insect trap and identification system (e.g., example insect imaging chambers and systems to process insect images) in the present disclosure can use a sophisticated camera system paired with novel software for insect detection and identification. The example insect trap is time-efficient, non-lethal, and user-friendly. In addition, the example insect imaging chamber in the present disclosure can be non-lethal traps for insect biodiversity monitoring. The example insect trap and identification system can preprocess collected images (video frames, films, photographs, etc.) to reduce the number of images and only use part of the image (bounding boxes) for insect trap and identification. Accordingly, the example insect trap and identification system can effectively and efficiently detect and classify insects using an AI system.--, in [0017]; and, -- [0034] The example insect trap and identification system can include two main components: an example insect imaging chamber and an example insect identification application. The insect trap and identification system can collect data. The example insect identification application can extract meaningful information from the data.
[0035] Hardware Stack: The example insect imaging chamber can represent a minimal baseline for collection of environmental and video data. The example insect imaging chamber can include a tube with two imaging devices (e.g., 12.3 megapixel HD cameras or any other suitable cameras) positioned such that they will capture the top and side views of insects as they crawl through the trap.--, in [0034]-[0035], --[0049] In further examples, the first cell 102 can include an insect guide tube 108 for placing the insect to be recorded by an imaging device, which can be disposed in the second cell 104. In some scenarios, the insect guide tube 108 can be removable. For example, the insect guide tube 108 can be removed from the imaging chamber 100 and replaced with a different insect guide tube 108 having a different size depending on expected subjects to be placed in the tube 108. In further examples, the insect guide tube 108 can be transparent such that the insect in the insect guide tube 108 can be seen from the outside of the insect guide tube 108 and recorded by an imaging device from the outside of the insect guide tube 108.--, in [0049]; and, --The second side image of the one or more insects can be different than the first side image captured from the first imaging device 122. For example, the first imaging device 122 can capture a top-view image of the insect based on the first reflector 120 while the second imaging device 130 can capture a side-view image of the insect based on the second reflector 128.--, in [0058]).
Re Claim 18, Patch as modified by DICK and YU further disclose wherein the capture zone of the trap assembly includes a porous base to enable dissemination of scent from a lure positioned underneath the capture zone (see Patch: e. g., Fig. 1, Fig.4, --[0017] Example intelligent insect trap and identification system (e.g., example insect imaging chambers and systems to process insect images) in the present disclosure can use a sophisticated camera system paired with novel software for insect detection and identification. The example insect trap is time-efficient, non-lethal, and user-friendly. In addition, the example insect imaging chamber in the present disclosure can be non-lethal traps for insect biodiversity monitoring. The example insect trap and identification system can preprocess collected images (video frames, films, photographs, etc.) to reduce the number of images and only use part of the image (bounding boxes) for insect trap and identification. Accordingly, the example insect trap and identification system can effectively and efficiently detect and classify insects using an AI system.--, in [0017]; and, -- [0034] The example insect trap and identification system can include two main components: an example insect imaging chamber and an example insect identification application. The insect trap and identification system can collect data. The example insect identification application can extract meaningful information from the data.
[0035] Hardware Stack: The example insect imaging chamber can represent a minimal baseline for collection of environmental and video data. The example insect imaging chamber can include a tube with two imaging devices (e.g., 12.3 megapixel HD cameras or any other suitable cameras) positioned such that they will capture the top and side views of insects as they crawl through the trap.--, in [0034]-[0035], --[0049] In further examples, the first cell 102 can include an insect guide tube 108 for placing the insect to be recorded by an imaging device, which can be disposed in the second cell 104. In some scenarios, the insect guide tube 108 can be removable. For example, the insect guide tube 108 can be removed from the imaging chamber 100 and replaced with a different insect guide tube 108 having a different size depending on expected subjects to be placed in the tube 108. In further examples, the insect guide tube 108 can be transparent such that the insect in the insect guide tube 108 can be seen from the outside of the insect guide tube 108 and recorded by an imaging device from the outside of the insect guide tube 108.--, in [0049]; and, --The second side image of the one or more insects can be different than the first side image captured from the first imaging device 122. For example, the first imaging device 122 can capture a top-view image of the insect based on the first reflector 120 while the second imaging device 130 can capture a side-view image of the insect based on the second reflector 128.--, in [0058]).
Re Claim 19, Patch as modified by DICK and YU further disclose a battery power supply, and wherein the image capture protocol and the data transmission protocol are configured to optimize battery power conservation (see Patch: e.g., -- the first imaging device 122 can be a digital camera, a video recording device, a camcorder, a motion picture camera; or any other suitable device capable of recording, storing, or transmitting visual images (e.g., photographs or videos) of insects. In addition, the first imaging device 122 can further include a motion sensor such that the first imaging device 122 records the visual images when the motion sensor detects movement of the insect in the elongated tube 114 to increase battery life if the first imaging device 122 is an battery powered device and save memory space by only storing insect images in the memory. In further examples, the first imaging device 122 or a controller can dynamically reduce a frame rate of the visual images (e.g., videos) to save battery usage.--, in [0057]).
Conclusion
Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEI WEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on 8:00 - 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662
Prosecution Insights