DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 3-4, 6-7, 10, & 14 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Eliyahu Benny (WIPO WO-2024/075121-A1, herein after “Benny”).
Regarding claim 1, Benny teaches [a] method for producing three-dimensional spectral models of a production system, the method comprising:
receiving input data (Page 25, “Multi-spectral images”) from a plurality of spectral markers (Page 25, “Targets”) disposed in the production system (Page 26, Agricultural setting or other relevant use case) “System 100 uses multi- spectral sensor 104 to generate the spectral data cube of the given observed area, to investigate and analyze the suspected potential targets using additional sensors and to identify potential targets within the spectral data cube all during the flight over the area, with no need to download the captured material for analysis in a ground station. This is achieved by system's 100 real-time analysis of the spectral data cube together with the input data coming from the additional sensors to better detect and identify the potential targets” (Benny, Page 25) in which “[a] non-limiting example is a real-time multi-spectral system 100 that is used in an agriculture setting. In this setting the target spectral signature can be of a given agriculture product that system 100 is trying to identify and further investigate” (Benny, Page 26);
generating the plurality of spectral markers (Page 6 “Targets”) within a three-dimensional space (Page 6, “Multi-spectral data cube”) based upon the input data “generate a multi-spectral data cube of a second FOV viewed by the multi- spectral sensor, utilizing the multi- spectral sensor, the calculated exposure times and the atmospheric correction matrix, wherein the generation of the multi- spectral data cube includes radiometric calibration and multi-channel registration; identify, utilizing the multi- spectral data cube, one or more potential targets, wherein each target is group of pixels identified within the multi-spectral data cube with a spectral signature that corresponds to at least one of the obtained target spectral signatures” (Benny, Page 6);
reading each of the spectral markers (Page 36, “Targets”) within the three-dimensional space to determine a unique spectral signature (Page 36, “Reflective spectral signature”) corresponding to each one of the spectral markers “multi- spectral system 100 can be configured to obtain: (A) a machine learning model capable of receiving the image of the scene and identify the existence of at least one object of the automatically identified objects within the scene, each identified object is associated with: (i) a location within the scene, and (ii) a predetermined representative reflective spectral signature, and (B) the image of the scene (block 1002)” (Benny, page 36) where each of the targets may be different materials at different positions within the 3D space, and appear a unique color depending on the respective material;
associating the determined spectral signatures of each of the plurality of spectral markers with a unique identification, wherein the unique identification corresponds to a location within the three-dimensional space “multi- spectral system 100 can be configured to obtain: (A) a machine learning model capable of receiving the image of the scene and identify the existence of at least one object of the automatically identified objects within the scene, each identified object is associated with: (i) a location within the scene, and (ii) a predetermined representative reflective spectral signature, and (B) the image of the scene (block 1002)” (Benny, Page 36) where the spectral signal is identified by its reflective color or location or a combination;
providing the unique identifications (Page 36, “Reflective spectral signature” and “location within the scene”)to a robot (Page 25, “Observation pod”) within the three-dimensional space, wherein the robot aligns itself within the three-dimensional space according to the associated locations “In accordance with the presently disclosed subject matter, the real-time multi- spectral system 100 can be optionally enclosed within an observation pod” (Benny, Page 25) where the observation pod may be attached to another device or location. Additionally, “System 100 is capable of keeping observation capabilities on a given observed area by controlling the movement of the multi-spectral sensor's 104 lens and of the lenses of the additional sensors (e.g., the wide daylight sensor 106, the narrow daylight sensor 110, the wide thermal sensor 114, the narrow thermal sensor 118, the narrow SWIR sensor 108, the laser rangefinder 116, the laser pointer 112, etc.) while the platform is in motion. In many cases the various sensors in pod or pay load, although they might have different FOVs, the center FOV of each sensor is aligned to the same direction, enabling the system to use different sensors for the same target location” (Benny, Page 25); and
reconstructing a three-dimensional spectral model including the assigned locations “System 100 can utilize the atmospheric simulator to generate more than one un-calibrated multi-spectral data cube with known target material from a single given calibrated multi-spectral data cube using different atmospheric conditions, thus allowing to easily create a large training data set from a small number of calibrated multi-spectral data cubes which are labeled with known target materials and their location within the scene” (Benny, Page 18) where the model may be labeled with their material and location.
Regarding claim 3, Benny teaches [t]he method of Claim 1, further comprising performing an action in response to the three-dimensional spectral model “System 100 is capable of keeping observation capabilities on a given observed area by controlling the movement of the multi-spectral sensor's 104 lens and of the lenses of the additional sensors (e.g., the wide daylight sensor 106, the narrow daylight sensor 110, the wide thermal sensor 114, the narrow thermal sensor 118, the narrow SWIR sensor 108, the laser rangefinder 116, the laser pointer 112, etc.) while the platform is in motion. In many cases the various sensors in pod or pay load, although they might have different FOVs, the center FOV of each sensor is aligned to the same direction, enabling the system to use different sensors for the same target location” (Benny, Page 25), the 3D model may also be “used for rendering game scenes by a game engine” (Benny, Page 22).
Regarding claim 4, Benny teaches [t]he method of Claim 1, wherein reading each of the spectral markers within the three-dimensional space (Page 26, “Spectral data cube”) to determine the unique spectral signature corresponding to each one of the spectral markers comprises:
reading a pattern (Page 26, “unique reflection”) of spectral values (Page 26, “Invisible wavelength”) that is unique to each one of the corresponding spectral markers (Page 26, “Target”) “Another example is of system 100 trying to identify special leaves (e.g., of artificial material) that are visually similar to all other leaves but are different in some invisible wavelength in which these special leaves have unique reflection which differs from regular leaves. In some cases, system 100 is not provided with specific target spectral signatures. In these cases, system 100 can determine if one or more anomalies exist in the spectral data cube of the given scene” (Benny, Page 26) where a pattern or unique invisible wavelength can be read from the target within the three-dimensional space.
Regarding claim 6, Benny teaches [t]he method of Claim 1, further comprising varying the spectral signature of at least one of the spectral markers over a period of time “The filter wheel is used to rotatably position a filter with a given wavelength range in the imaging path of the sensor to capture an image in that wavelength range. The filter in the imaging path of the sensor at a given time- window and/or interval of time is the active filter at that time window” (Benny, Page 16), wherein varying the spectral signature comprises:
cyclically or non-cyclically varying the spectral signature “Multi- spectral sensor 104 can alternatively comprise two or more filter wheels, which can be used to combine different filters from each of the filter wheels simultaneously. In addition, multi- spectral sensor 104 can alternatively utilize one or more of the following filter methods: (a) a Fabry-Perot interferometer (FPI), (b) a Linear Variable filter (LVF), and (c) a circular variable filter. Multi- spectral sensor 104 can be alternatively comprised of multiple sensors, each sensor captures images of the scene at a different wavelength range, or a combination of multiple sensors with filters enabling each sensor to capture different wavelength ranges” (Benny, Page 16) where using a different filter or method of reading the target is considered varying the observed spectral signature;
varying a wave amplitude or a frequency of the spectral marker “In this setting the target spectral signature can be of a given agriculture product that system 100 is trying to identify and further investigate. For example, a spectral signature of a strawberry. The spectral signature of the strawberry is a vector associated with multiple values. Each value in this vector is associated with the spectral variation of reflectance or emittance of a strawberry in the wavelength ranges that can be captured by the multi-spectral sensor 104” (Benny, Page 26) where over a period of time spectral signatures of agricultural products can be changed and observed, additionally “The filter wheel is used to rotatably position a filter with a given wavelength range in the imaging path of the sensor to capture an image in that wavelength range. The filter in the imaging path of the sensor at a given time- window and/or interval of time is the active filter at that time window. The wheel is then turned to position the next filter, which has a different wavelength range in the imaging path of the sensor allowing the sensor to capture an image in the different wavelength range. Rotating the filter wheel allows the multi- spectral sensor 104 to capture a series of images in multiple wavelength ranges over a given time-frame.” (Benny, Page 16).
Regarding claim 7, Benny teaches [t]he method of Claim 1, further comprising transmitting a supplemental data signal (Benny, Page 25 “thermal sensor” information) from at least one of the spectral markers “System 100 is capable of keeping observation capabilities on a given observed area by controlling the movement of the multi-spectral sensor's 104 lens and of the lenses of the additional sensors (e.g., the wide daylight sensor 106, the narrow daylight sensor 110, the wide thermal sensor 114, the narrow thermal sensor 118, the narrow SWIR sensor 108, the laser rangefinder 116, the laser pointer 112, etc.) while the platform is in motion. In many cases the various sensors in pod or pay load, although they might have different FOVs, the center FOV of each sensor is aligned to the same direction, enabling the system to use different sensors for the same target location” (Benny, Page 25) where the thermal sensor receives information about temperature, wherein the supplemental data signal is comprised of at least one signal received from the production system “This is achieved by system's 100 real-time analysis of the spectral data cube together with the input data coming from the additional sensors to better detect and identify the potential targets” (Benny, Page 25).
Regarding claim 10, Benny teaches [t]he method of Claim 1, wherein reading each of the spectral markers within the three-dimensional space to determine the unique spectral signature corresponding to each one of the spectral markers comprises reading the spectral markers with a spectral device configured to read a pattern (Page 26, “unique reflection”) of spectral values of each spectral signature, wherein the spectral device is disposed on the robot “In accordance with the presently disclosed subject matter, the real-time multi- spectral system 100 can be optionally enclosed within an observation pod. The observation pod can be used as a stand-alone system and/or can be installed as a special purpose payload on a platform, for example: on a personal platform, on a watch tower as part of a wider border defense system, on a ground vehicle, on an aerial platform, in space, etc. When system 100 is installed in an observation pod on an aerial platform, it can detect potential targets in a given scene in real-time, without the need to download information from the aerial platform to a ground station for human operator analysis. System 100 is capable of keeping observation capabilities on a given observed area by controlling the movement of the multi-spectral sensor's 104 lens and of the lenses of the additional sensors (e.g., the wide daylight sensor 106, the narrow daylight sensor 110, the wide thermal sensor 114, the narrow thermal sensor 118, the narrow SWIR sensor 108, the laser rangefinder 116, the laser pointer 112, etc.) while the platform is in motion.” (Benny, Page 25) where system 100 can detect new or predetermined patterns of spectral values using the various sensors disposed on the robot which could be a vehicle or other tool.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Benny in view of Mark Profio (Pat. Pub. US-20110295113, herein after “Profio”).
Regarding claim 2, Benny teaches [t]he method of Claim 1,
Benny does not explicitly teach displaying the reconstructed three-dimensional volume.
Profio teaches displaying the reconstructed three-dimensional volume “Thereafter, at 86, three-dimensional (3D) volume visualization is performed on the 2D stacks of images to form 3D volume visualization models. In particular, maximum intensity pixel projection is performed through each stack of 2D images resulting in a 3D MIP volume at each point in time.” (Profio, ¶ [027]) where taking 2D splices of an object using x-ray methods, then combining them produces a 3D model with volume.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of producing 3-dimensional spectral models taught by Benny with the display of volume taught by Profio to produce a 3-dimensional spectral model display that included volume. The motivation to do so would be to visualize a scale of the model.
Claims 5 & 8 are rejected under 35 U.S.C. 103 as being unpatentable over Benny in view of Brian Brogger Et. Al. (Pat. Pub. US-20220334054-A1, herein after “Brogger”).
Regarding claim 5, Benny teaches [t]he method of Claim 4, wherein reading the pattern (Benny, Page 26, “unique reflection”) of spectral values (Benny, Page 26, “Invisible wavelength”) that is unique to each one of the corresponding spectral markers (Benny, Page 26, “Target”) comprises:
reading a first area comprising a spectral value equal to a spectral value corresponding to the three-dimensional space “a multi- spectral potential target identification system, the system comprising: a multi- spectral sensor capable of acquiring images in a plurality of imaging channels, each having a different wavelength range; one or more additional sensors; and a processing circuitry configured to: obtain one or more target spectral signatures” (Benny, Page 6) where the targets are located in a three-dimensional space;
Benny fails to explicitly teach reading at least one neutral area;
reading a second area comprising a spectral value which is higher relative to the first area;
reading a third area comprising a spectral value which is lower relative to first area, wherein the first, second, and third areas of each of the spectral markers are arranged in a surface pattern that is unique to each one of the corresponding spectral markers; and
providing a contrast between at least two of the areas forming the pattern of spectral values that is unique to each one of the corresponding spectral markers.
Brogger teaches reading at least one neutral area “In various embodiments, the opaque base layers and the spectral taggant layers may be formulated in various ways. For example, opaque base layers useful in any embodiments of the present invention are often formulated to provide a polymer matrix presenting a single, neutral, opaque, color such as an opaque white or grey, but these can be formulated to display one or more other colors or other surface characteristics, if desired” (Brogger, ¶ [0074]);
reading a second area comprising a spectral value which is higher relative to the first area “FIG. 3 shows an alternative embodiment of a platelet-shaped, one-sided taggant particle 34. In a manner similar to taggant particles 10 of FIGS. 1 and 2, taggant particle 34 also has opposed and parallel major faces 35 and a side 37 that interconnect the major faces 35 around the perimeter of the major faces 35. Using an illustrative manufacturing method described further below, the major faces 35 are generally parallel to each other so that the height of the sides 37 between the major faces 35 is generally uniform. The method may have a tendency to produce taggant particles 34 for which the perimeters 39 defining the major face shapes are somewhat irregular. In contrast to taggant particle 10 of FIGS. 1 and 2, taggant particle 34 deploys taggant material in a stack of multiple spectral taggant layers 38, 44, and 50 provided on opaque base layer 36” (Brogger, ¶ [0066]) where the taggant particles may vary in spectral value (i.e. higher or lower) “Different taggant particles may be used in combination to produce even more complex signatures.” (Brogger, ¶ [0013]);
reading a third area comprising a spectral value which is lower relative to first area, wherein the first, second, and third areas of each of the spectral markers are arranged in a surface pattern that is unique to each one of the corresponding spectral markers “Taggant system 26 generally includes one or more taggants. For purposes of illustration, taggant system 26 includes a combination of taggants 28 and 30 incorporated into the same spectral taggant layer 24. In other modes of practice, each of taggants 28 and 30 could be incorporated into separate spectral taggant layers if desired. Using a combination of two or more spectral taggants 28 and 30 offers many signatures strategies to be implemented” (Brogger, ¶ 0062]); and
PNG
media_image1.png
553
647
media_image1.png
Greyscale
Brogger, Figs. 1 & 2 show the taggant layer (item 24) which reads upon the area of spectral markers and opaque layer (item 22) that reads upon the contrasting neutral zone.
PNG
media_image2.png
232
629
media_image2.png
Greyscale
Brogger, Fig. 3, where different spectral taggant layers (items 38, 44, and 50) which read upon the multiple different layers are placed on an opaque surface (item 36) which reads upon the neutral zone.
providing a contrast between at least two of the areas forming the pattern of spectral values that is unique to each one of the corresponding spectral markers “The taggant particles of the present invention such as those illustrated in FIGS. 1 to 6 may be manufactured using a variety of different methods. According to a preferred approach, the taggant particles of the present invention such as those described in FIGS. 1-6 may be manufactured using a three-stage process. In a first stage, a multilayer sheet of substantially uniform thickness is prepared. The layer stack in the sheet corresponds to the sequence of layers in the desired taggant particles. For example, to prepare a sheet corresponding to the taggant particles 10 of FIGS. 1 and 2, a layer stack would include a layer corresponding to opaque base layer 22 and spectral taggant layer 24. The sheet may be formed with either layer 22 supporting layer 24 or vice versa. As another example, to prepare a sheet corresponding to taggant particle 90 of FIG. 6, a multilayer sheet would be prepared that has a sequence of layers stacked in a manner corresponding to the layer stack of spectral taggant layers 92, 98, 104, 112, and 118 and opaque base layers 124 in taggant particles 90.” (Brogger, ¶ [0109]) where different taggants may be used on each layer with an opaque base to provide contrast.
PNG
media_image3.png
348
649
media_image3.png
Greyscale
Brogger, Fig. 6, multilayer sheet with sequence of spectral taggant layers (items 92, 98, and 104) which may be organized in any desired pattern.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of producing 3-dimensional spectral models taught by Benny with the areas of different spectral values taught by Brogger to produce a unique surface pattern that included multiple varied spectral signatures. The motivation to do so would be to discern different signatures “A composite signature, therefore, is more complex and more unique to make it easier to distinguish, harder to reverse engineer, able to encode more information, and/or the like” (Brogger, ¶ [0157]).
Regarding claim 8, Benny in view of Brogger teach [t]he method of Claim 5, further comprising masking the at least one neutral area, the first area, the second area, or the third area with a spectral mask (Benny, Page 23, (objects) masks) “automatically generate hyperspectral and/or multi-spectral training data-sets are depicted in Fig. 15. At step 1502 a Materials database that stores a reflectance spectrum, a label (materials name), and a representative color for at least one type of material is used together with one or more heuristics tables from step 1504 and a photo-realistic engine of a 3D virtual world (such as: a game engine like the unreal engine, etc.) which includes RGB frames of the virtual world, distance maps within the virtual world, colors maps of scenes and objects within the virtual world and labels (objects) masks (from step 1506) to transplant materials into the virtual world” (Benny, Page 23).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Benny in view of Karl Garsha (Pat. Pub. JP-2021047207-A, herein after “Garsha”).
Regarding claim 9, Benny teaches [t]he method of Claim 1,
Benny fails to teach further comprising maintaining at least a portion of each of the spectral markers at an invariant spectral value, wherein maintaining at least a portion of each of the spectral markers at an invariant spectral value comprises powering the at least one portion of the spectral markers from a power source within the three-dimensional space or from an outside or independent power source.
Garsha teaches further comprising maintaining at least a portion of each of the spectral markers at an invariant spectral value, wherein maintaining at least a portion of each of the spectral markers at an invariant spectral value comprises powering the at least one portion of the spectral markers from a power source within the three-dimensional space or from an outside or independent power source “The power and temperature of the spectrum source 410 may be stabilized and monitored by closed-loop electronics and / or multi-bandpass filters 410a. The multi-bandpass filter 410a has n predetermined passbands and is positioned in front of the spectrum source 410. In an example embodiment of the invention, a spectrum acquisition system 422, eg, a microscope-based light acquisition device, includes or is coupled to a spectrum camera 443. The spectrum acquisition device 442 includes a scanning platform 445” (Garsha, Page 11) where maintaining a stable power of the spectrum source outputs a maintained spectrum.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of producing 3-dimensional spectral models taught by Benny with the use of maintaining a constant spectral source taught by Garsha to produce a reference spectral output. The motivation to do so would be to compare the constant spectral output with other spectral signatures to identify differences.
Claims 11-12, & 14 are rejected under 35 U.S.C. 103 as being unpatentable over Benny in view of Profio, and further in view of Garsha.
Regarding claim 11, Benny teaches [a] computing system, comprising:
one or more processors “Multi- spectral system 100 further comprises processing circuitry 102” (Benny, Page 24);
a plurality of spectral markers communicated to the one or more processors, wherein the plurality of spectral markers are disposed in a production system “The capture can be for example during a flight of a platform to which multi-spectral sensor 104 is attached. A frame is captured per wavelength of the sensor. Each frame has a different angle towards the scene and targets within the scene due to the movement of the platform over the scene while capturing the frames” (Benny, Page 22);
a robot communicated to the one or more processors, wherein the robot is configured to read the plurality of spectral markers “The observation pod can be used as a stand-alone system and/or can be installed as a special purpose payload on a platform, for example: on a personal platform, on a watch tower as part of a wider border defense system, on a ground vehicle, on an aerial platform, in space, etc. When system 100 is installed in an observation pod on an aerial platform, it can detect potential targets in a given scene in real-time, without the need to download information from the aerial platform to a ground station for human operator analysis” (Benny, Page 25);
at least one sensor communicated to the one or more processors “System 100 is capable of keeping observation capabilities on a given observed area by controlling the movement of the multi-spectral sensor's 104 lens and of the lenses of the additional sensors (e.g., the wide daylight sensor 106, the narrow daylight sensor 110, the wide thermal sensor 114, the narrow thermal sensor 118, the narrow SWIR sensor 108, the laser rangefinder 116, the laser pointer 112, etc.) while the platform is in motion” (Benny, Page 25); and
a memory system comprising one or more non-transitory computer-readable media storing instructions that, when executed by at least one of the one or more processors, cause the computing system to perform operations, the operations comprising “The processing circuitry 102 can read the relevant parameters of each algorithm from a first designated memory area of system 100, read the spectral data cube, preform the algorithms on the spectral data cube's images and save the results in a second designated memory area of system 100” (Benny, Page 28):
receiving input data from the plurality of spectral markers, the input data representing the production system “System 100 uses multi- spectral sensor 104 to generate the spectral data cube of the given observed area, to investigate and analyze the suspected potential targets using additional sensors and to identify potential targets within the spectral data cube all during the flight over the area, with no need to download the captured material for analysis in a ground station. This is achieved by system's 100 real-time analysis of the spectral data cube together with the input data coming from the additional sensors to better detect and identify the potential targets” (Benny, Page 25) in which “[a] non-limiting example is a real-time multi-spectral system 100 that is used in an agriculture setting. In this setting the target spectral signature can be of a given agriculture product that system 100 is trying to identify and further investigate” (Benny, Page 26);
generating the plurality of spectral markers within a three-dimensional space based upon the input data “generate a multi-spectral data cube of a second FOV viewed by the multi- spectral sensor, utilizing the multi- spectral sensor, the calculated exposure times and the atmospheric correction matrix, wherein the generation of the multi- spectral data cube includes radiometric calibration and multi-channel registration; identify, utilizing the multi- spectral data cube, one or more potential targets, wherein each target is group of pixels identified within the multi-spectral data cube with a spectral signature that corresponds to at least one of the obtained target spectral signatures” (Benny, Page 6);
reading each of the spectral markers within the three-dimensional space to determine a unique spectral signature corresponding to each one of the spectral markers, wherein reading the plurality of spectral markers comprises reading the spectral markers with a spectral device configured to read a pattern of spectral values of each spectral signature, and wherein the spectral device is disposed on the robot “In accordance with the presently disclosed subject matter, the real-time multi- spectral system 100 can be optionally enclosed within an observation pod. The observation pod can be used as a stand-alone system and/or can be installed as a special purpose payload on a platform, for example: on a personal platform, on a watch tower as part of a wider border defense system, on a ground vehicle, on an aerial platform, in space, etc. When system 100 is installed in an observation pod on an aerial platform, it can detect potential targets in a given scene in real-time, without the need to download information from the aerial platform to a ground station for human operator analysis. System 100 is capable of keeping observation capabilities on a given observed area by controlling the movement of the multi-spectral sensor's 104 lens and of the lenses of the additional sensors (e.g., the wide daylight sensor 106, the narrow daylight sensor 110, the wide thermal sensor 114, the narrow thermal sensor 118, the narrow SWIR sensor 108, the laser rangefinder 116, the laser pointer 112, etc.) while the platform is in motion.” (Benny, Page 25) where system 100 can detect new or predetermined patterns of spectral values using the various sensors disposed on the robot which could be a vehicle or other tool;
associating the determined spectral signatures of each of the plurality of spectral markers with a unique identification, wherein the unique identification corresponds to a location within the three-dimensional space “multi- spectral system 100 can be configured to obtain: (A) a machine learning model capable of receiving the image of the scene and identify the existence of at least one object of the automatically identified objects within the scene, each identified object is associated with: (i) a location within the scene, and (ii) a predetermined representative reflective spectral signature, and (B) the image of the scene (block 1002)” (Benny, Page 36) where the spectral signal is identified by its reflective color or location or a combination;
providing the unique identifications to the robot within the three-dimensional space, wherein the robot aligns itself within the three-dimensional space according to the associated locations “In accordance with the presently disclosed subject matter, the real-time multi- spectral system 100 can be optionally enclosed within an observation pod” (Benny, Page 25) where the observation pod may be attached to another device or location. Additionally, “System 100 is capable of keeping observation capabilities on a given observed area by controlling the movement of the multi-spectral sensor's 104 lens and of the lenses of the additional sensors (e.g., the wide daylight sensor 106, the narrow daylight sensor 110, the wide thermal sensor 114, the narrow thermal sensor 118, the narrow SWIR sensor 108, the laser rangefinder 116, the laser pointer 112, etc.) while the platform is in motion. In many cases the various sensors in pod or pay load, although they might have different FOVs, the center FOV of each sensor is aligned to the same direction, enabling the system to use different sensors for the same target location” (Benny, Page 25);
reconstructing a three-dimensional spectral model using the robot or the at least one sensor, wherein the three-dimensional spectral model comprises a three-dimensional volume including the assigned locations “System 100 can utilize the atmospheric simulator to generate more than one un-calibrated multi-spectral data cube with known target material from a single given calibrated multi-spectral data cube using different atmospheric conditions, thus allowing to easily create a large training data set from a small number of calibrated multi-spectral data cubes which are labeled with known target materials and their location within the scene” (Benny, Page 18) where the model is a data cube and may be labeled with the material and location; and
Benny does not teach maintaining at least a portion of each of the spectral markers at an invariant spectral value, wherein the spectral markers receive power to maintain the at least one portion at the invariant spectral value from the three-dimensional space or from an outside or independent power source;
Nor does Benny teach displaying the reconstructed three-dimensional volume.
Garsha teaches maintaining at least a portion of each of the spectral markers at an invariant spectral value, wherein the spectral markers receive power to maintain the at least one portion at the invariant spectral value from the three-dimensional space or from an outside or independent power source “The power and temperature of the spectrum source 410 may be stabilized and monitored by closed-loop electronics and / or multi-bandpass filters 410a. The multi-bandpass filter 410a has n predetermined passbands and is positioned in front of the spectrum source 410. In an example embodiment of the invention, a spectrum acquisition system 422, eg, a microscope-based light acquisition device, includes or is coupled to a spectrum camera 443. The spectrum acquisition device 442 includes a scanning platform 445” (Garsha, Page 11) where maintaining a stable power of the spectrum source outputs a maintained spectrum;
Profio teaches displaying the reconstructed three-dimensional volume “Thereafter, at 86, three-dimensional (3D) volume visualization is performed on the 2D stacks of images to form 3D volume visualization models. In particular, maximum intensity pixel projection is performed through each stack of 2D images resulting in a 3D MIP volume at each point in time.” (Profio, ¶ [027]) where taking 2D splices of an object using x-ray methods, then combining them produces a 3D model with volume.
Regarding claim 12, Benny in view of Profio and Garsha teach [t]he computing system of Claim 11, wherein the unique spectral signature of each of the spectral markers comprises a wavelength between 100nm and 15mm “In accordance with the presently disclosed subject matter, the real-time multi- spectral system 100 (also interchangeably referred to herein as "multi- spectral system 100" or as "system 100") can comprise a multi- spectral sensor 104 capable of capturing image data within specific wavelength ranges across the electromagnetic spectrum, for example: in visible (VIS) wavelengths, in ultraviolet (UV) wavelength, in Short-Wave Infrared (SWIR) wavelengths, in Near Infrared (NIR) wavelengths, in Middle Wavelength Infrared (MWIR) wavelengths, in Long Wavelength Infrared (LWIR) wavelengths, etc., or in any combination thereof” (Benny, Page 16) where it is known that UV, visible, and IR light have wavelengths on the electromagnetic spectrum between 100nm and 15mm, wherein each of the spectral markers comprises a two-dimensional or three-dimensional shape “… a machine learning model capable of receiving: (a) a source 2D image of the 2D images of the scene, and (b) a target 2D image of the 2D images of the scene, and determining a corresponding flow map mapping changes to be made to pixels of the source 2D image in order to align the source 2D image with the target 2D image …” (Benny, Page 37) Where the target image is a two-dimensional representation of the target spectral signatures, which may be 2D, or 3D, wherein each of the spectral signatures comprises a pattern of spectral values that is unique to each one of the corresponding spectral markers “Another example is of system 100 trying to identify special leaves (e.g., of artificial material) that are visually similar to all other leaves but are different in some invisible wavelength in which these special leaves have unique reflection which differs from regular leaves. In some cases, system 100 is not provided with specific target spectral signatures. In these cases, system 100 can determine if one or more anomalies exist in the spectral data cube of the given scene” (Benny, Page 26) where a pattern or unique invisible wavelength can be read from the target within the three-dimensional space, and wherein each of the spectral markers comprises at least one portion that is reflective “each identified object is associated with: (i) a location within the scene, and (ii) a predetermined representative reflective spectral signature” (Benny, Page 36).
Regarding claim 14, Benny in view of Profio and Garsha teach [t]he computing system of Claim 11, wherein the operations performed by the computing system further comprises varying the spectral signature of at least one of the spectral markers over a period of time “The filter wheel is used to rotatably position a filter with a given wavelength range in the imaging path of the sensor to capture an image in that wavelength range. The filter in the imaging path of the sensor at a given time- window and/or interval of time is the active filter at that time window” (Benny, Page 16), wherein varying the spectral signature comprises: cyclically or non-cyclically varying the spectral signature “Multi- spectral sensor 104 can alternatively comprise two or more filter wheels, which can be used to combine different filters from each of the filter wheels simultaneously. In addition, multi- spectral sensor 104 can alternatively utilize one or more of the following filter methods: (a) a Fabry-Perot interferometer (FPI), (b) a Linear Variable filter (LVF), and (c) a circular variable filter. Multi- spectral sensor 104 can be alternatively comprised of multiple sensors, each sensor captures images of the scene at a different wavelength range, or a combination of multiple sensors with filters enabling each sensor to capture different wavelength ranges” (Benny, Page 16) where using a different filter or method of reading the target is considered varying the observed spectral signature; varying a wave amplitude or a frequency of the spectral marker “In this setting the target spectral signature can be of a given agriculture product that system 100 is trying to identify and further investigate. For example, a spectral signature of a strawberry. The spectral signature of the strawberry is a vector associated with multiple values. Each value in this vector is associated with the spectral variation of reflectance or emittance of a strawberry in the wavelength ranges that can be captured by the multi-spectral sensor 104” (Benny, Page 26) where over a period of time spectral signatures of agricultural products can be changed and observed, additionally “The filter wheel is used to rotatably position a filter with a given wavelength range in the imaging path of the sensor to capture an image in that wavelength range. The filter in the imaging path of the sensor at a given time- window and/or interval of time is the active filter at that time window. The wheel is then turned to position the next filter, which has a different wavelength range in the imaging path of the sensor allowing the sensor to capture an image in the different wavelength range. Rotating the filter wheel allows the multi- spectral sensor 104 to capture a series of images in multiple wavelength ranges over a given time-frame.” (Benny, Page 16).
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Benny in view of Profio, Garsha, and further in view of Brogger.
Regarding claim 13, Benny in view of Profio and Garsha teach [t]he computing system of Claim 12, wherein the pattern (Benny, Page 26, “unique reflection”) of spectral values (Benny, Page 26, “Invisible wavelength”) corresponding to each of the spectral markers (Benny, Page 26, “Target”) comprises:
a first area comprising a spectral value equal to the three-dimensional space “a multi- spectral potential target identification system, the system comprising: a multi- spectral sensor capable of acquiring images in a plurality of imaging channels, each having a different wavelength range; one or more additional sensors; and a processing circuitry configured to: obtain one or more target spectral signatures” (Benny, Page 6) where the targets are located in a three-dimensional space and have an equal respective spectral value;
Benny in view of Profio, Garsha and further in view of Brogger fail to explicitly teach at least one neutral area;
a second area comprising a spectral value which is higher relative to the first area; and
a third area comprising a spectral value which is lower relative to first area, wherein the first, second, and third areas of each of the spectral markers are arranged in a surface pattern that is unique to each one of the corresponding spectral markers, wherein the second area comprises a spectral value which is higher relative to a spectral value of the at least one neutral area, and wherein the third area comprises a spectral value which is lower relative to the spectral value of the at least one neutral area, wherein the pattern of spectral values that is unique to each one of the corresponding spectral markers is configured to provide a contrast between at least two areas of the spectral marker, and wherein the at least one neutral area is comprised of a material configured to provide a contrast with the first, second, or third area.
Brogger teaches at least one neutral area “In various embodiments, the opaque base layers and the spectral taggant layers may be formulated in various ways. For example, opaque base layers useful in any embodiments of the present invention are often formulated to provide a polymer matrix presenting a single, neutral, opaque, color such as an opaque white or grey, but these can be formulated to display one or more other colors or other surface characteristics, if desired” (Brogger, ¶ [0074]);
a second area comprising a spectral value which is higher relative to the first area “FIG. 3 shows an alternative embodiment of a platelet-shaped, one-sided taggant particle 34. In a manner similar to taggant particles 10 of FIGS. 1 and 2, taggant particle 34 also has opposed and parallel major faces 35 and a side 37 that interconnect the major faces 35 around the perimeter of the major faces 35. Using an illustrative manufacturing method described further below, the major faces 35 are generally parallel to each other so that the height of the sides 37 between the major faces 35 is generally uniform. The method may have a tendency to produce taggant particles 34 for which the perimeters 39 defining the major face shapes are somewhat irregular. In contrast to taggant particle 10 of FIGS. 1 and 2, taggant particle 34 deploys taggant material in a stack of multiple spectral taggant layers 38, 44, and 50 provided on opaque base layer 36” (Brogger, ¶ [0066]) where the taggant particles may vary in spectral value (i.e. higher or lower) “Different taggant particles may be used in combination to produce even more complex signatures.” (Brogger, ¶ [0013]); and
a third area comprising a spectral value which is lower relative to first area, wherein the first, second, and third areas of each of the spectral markers are arranged in a surface pattern that is unique to each one of the corresponding spectral markers “Taggant system 26 generally includes one or more taggants. For purposes of illustration, taggant system 26 includes a combination of taggants 28 and 30 incorporated into the same spectral taggant layer 24. In other modes of practice, each of taggants 28 and 30 could be incorporated into separate spectral taggant layers if desired. Using a combination of two or more spectral taggants 28 and 30 offers many signatures strategies to be implemented” (Brogger, ¶ 0062]), wherein the second area comprises a spectral value which is higher relative to a spectral value of the at least one neutral area, and wherein the third area comprises a spectral value which is lower relative to the spectral value of the at least one neutral area “In contrast to taggant particle 10 of FIGS. 1 and 2, taggant particle 34 deploys taggant material in a stack of multiple spectral taggant layers 38, 44, and 50 provided on opaque base layer 36” (Brogger, ¶ [0066]) where the taggant particles may vary in spectral value (i.e. higher or lower) and have contrast to the opaque base layer, or neutral area “Different taggant particles may be used in combination to produce even more complex signatures.” (Brogger, ¶ [0013]), wherein the pattern of spectral values that is unique to each one of the corresponding spectral markers is configured to provide a contrast between at least two areas of the spectral marker “Fig. 6 shows an alternative embodiment of a platelet-shaped, two-sided taggant particle 90 that incorporates different spectral signatures on each side. To accomplish this, spectral taggant layers … are deployed on one side of opaque base layers 124. In the meantime, spectral taggant layers … are deployed on the other side of opaque base layers 124. The result is that each side of taggant particle 90 encodes a different spectral signature” (Brogger, ¶ [0055]), and wherein the at least one neutral area is comprised of a material configured to provide a contrast with the first, second, or third area “The solid background helps to allow a better, stronger spectral signal to be read, particularly when the spectral signature is read remotely from a distance. Also important, the solid background helps to produce a consistent spectral output that is less vulnerable to substrate color, translucency, ambient light, and other background noise that could affect reading the output, in comparison tests, spectral signatures of particle embodiments including one or more base color layers would be easily read from a distance using multispectral/hyperspectral imaging techniques” (Brogger, ¶ [0045]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of producing 3-dimensional spectral models taught by Benny with the areas of different spectral values taught by Brogger to produce a unique surface pattern that included multiple varied spectral signatures and neutral areas for contrast. The motivation to do so would be to discern different signatures and produce unique signatures “A composite signature, therefore, is more complex and more unique to make it easier to distinguish, harder to reverse engineer, able to encode more information, and/or the like” (Brogger, ¶ [0157]).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Benny in view of Profio, Garsha, and further in view of Weimin Zhang (Pat. Pub. US-20190113445, herein after “Zhang”).
Regarding claim 15, Benny in view of Profio and Garsha teach [t]he computer system of Claim 11,
Benny in view of Profio and Garsha fail to explicitly teach wherein the operations performed by the computing system further comprises transmitting a supplemental data signal from at least one of the spectral markers to the one or more processors, wherein the supplemental data signal is comprised of at least one of the following: GPS data, humidity, detection or concentration of a gas, pressure, fluid level, or a combination thereof.
Zhang teaches wherein the operations performed by the computing system further comprises transmitting a supplemental data signal from at least one of the spectral markers to the one or more processors, wherein the supplemental data signal is comprised of at least one of the following: GPS data, humidity, detection or concentration of a gas, pressure, fluid level, or a combination thereof “Each infrared emission host 111 emits infrared rays to a corresponding infrared reflective assembly 112 to monitor the air mass in the monitored range of the monitoring equipment 11 in real time to obtain the infrared spectral data of the chemical factors in the passing air mass. The pollution factor qualitative module 12 presets the infrared judgment reference data of a plurality of pollution factors, which uses the infrared spectrum values of a plurality of different pollution factors as a reference” (Zhang, ¶ [0041]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of producing 3-dimensional spectral models taught by Benny with the transmission of certain data from the spectral marker taught by Zhang to read information directly from the spectral marker. The motivation to do so would be to analyze the status of the spectral marker or its surroundings, or to better communicate with the marker.
Claims 16-20 are rejected under 35 U.S.C. 103 as being unpatentable over Benny in view of Brogger, Zhang, Garsha, Profio, and further in view of Ahmed Aqrawi Et. Al. (Pat. Pub. WO-2017168191-A1, herein after “Aqrawi”).
Regarding claim 16, Benny teaches [a] non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations, the operations comprising:
receiving input data representing a production system “System 100 uses multi- spectral sensor 104 to generate the spectral data cube of the given observed area, to investigate and analyze the suspected potential targets using additional sensors and to identify potential targets within the spectral data cube all during the flight over the area, with no need to download the captured material for analysis in a ground station. This is achieved by system's 100 real-time analysis of the spectral data cube together with the input data coming from the additional sensors to better detect and identify the potential targets” (Benny, Page 25) in which “[a] non-limiting example is a real-time multi-spectral system 100 that is used in an agriculture setting. In this setting the target spectral signature can be of a given agriculture product that system 100 is trying to identify and further investigate” (Benny, Page 26);
generating a plurality of spectral markers based upon the input data, wherein the spectral markers are generated within a three-dimensional space “generate a multi-spectral data cube of a second FOV viewed by the multi- spectral sensor, utilizing the multi- spectral sensor, the calculated exposure times and the atmospheric correction matrix, wherein the generation of the multi- spectral data cube includes radiometric calibration and multi-channel registration; identify, utilizing the multi- spectral data cube, one or more potential targets, wherein each target is group of pixels identified within the multi-spectral data cube with a spectral signature that corresponds to at least one of the obtained target spectral signatures” (Benny, Page 6), wherein each of the spectral markers comprises a unique spectral signature “multi- spectral system 100 can be configured to obtain: (A) a machine learning model capable of receiving the image of the scene and identify the existence of at least one object of the automatically identified objects within the scene, each identified object is associated with: (i) a location within the scene, and (ii) a predetermined representative reflective spectral signature, and (B) the image of the scene (block 1002)” (Benny, page 36) where each of the targets may be different materials at different positions within the 3D space, and appear a unique color depending on the respective material, wherein the unique spectral signature of each of the spectral markers comprises a wavelength between 100nm and 15mm “In accordance with the presently disclosed subject matter, the real-time multi- spectral system 100 (also interchangeably referred to herein as "multi- spectral system 100" or as "system 100") can comprise a multi- spectral sensor 104 capable of capturing image data within specific wavelength ranges across the electromagnetic spectrum, for example: in visible (VIS) wavelengths, in ultraviolet (UV) wavelength, in Short-Wave Infrared (SWIR) wavelengths, in Near Infrared (NIR) wavelengths, in Middle Wavelength Infrared (MWIR) wavelengths, in Long Wavelength Infrared (LWIR) wavelengths, etc., or in any combination thereof” (Benny, Page 16) where it is known that UV, visible, and IR light have wavelengths on the electromagnetic spectrum between 100nm and 15mm, wherein each of the spectral markers comprises a two-dimensional or three-dimensional shape “… a machine learning model capable of receiving: (a) a source 2D image of the 2D images of the scene, and (b) a target 2D image of the 2D images of the scene, and determining a corresponding flow map mapping changes to be made to pixels of the source 2D image in order to align the source 2D image with the target 2D image …” (Benny, Page 37) Where the target image is a two-dimensional representation of the target spectral signatures, which may be 2D, or 3D, wherein each of the spectral signatures comprises a pattern (Benny, Page 26, “unique reflection”) of spectral values (Benny, Page 26, “Invisible wavelength”) that is unique to each one of the corresponding spectral (Benny, Page 26, “Target”) markers “Another example is of system 100 trying to identify special leaves (e.g., of artificial material) that are visually similar to all other leaves but are different in some invisible wavelength in which these special leaves have unique reflection which differs from regular leaves. In some cases, system 100 is not provided with specific target spectral signatures. In these cases, system 100 can determine if one or more anomalies exist in the spectral data cube of the given scene” (Benny, Page 26) where a pattern or unique invisible wavelength can be read from the target within the three-dimensional space, wherein each of the spectral markers comprises at least one portion that is reflective “each identified object is associated with: (i) a location within the scene, and (ii) a predetermined representative reflective spectral signature” (Benny, Page 36), wherein the pattern of each of the spectral markers comprises:
a first area comprising a spectral value equal to the three-dimensional space “a multi- spectral potential target identification system, the system comprising: a multi- spectral sensor capable of acquiring images in a plurality of imaging channels, each having a different wavelength range; one or more additional sensors; and a processing circuitry configured to: obtain one or more target spectral signatures” (Benny, Page 6) where the targets are located in a three-dimensional space;
varying the spectral signature of at least one of the spectral markers over a period of time “The filter wheel is used to rotatably position a filter with a given wavelength range in the imaging path of the sensor to capture an image in that wavelength range. The filter in the imaging path of the sensor at a given time- window and/or interval of time is the active filter at that time window” (Benny, Page 16), wherein varying the spectral signature comprises:
cyclically or non-cyclically varying the spectral signature “Multi- spectral sensor 104 can alternatively comprise two or more filter wheels, which can be used to combine different filters from each of the filter wheels simultaneously. In addition, multi- spectral sensor 104 can alternatively utilize one or more of the following filter methods: (a) a Fabry-Perot interferometer (FPI), (b) a Linear Variable filter (LVF), and (c) a circular variable filter. Multi- spectral sensor 104 can be alternatively comprised of multiple sensors, each sensor captures images of the scene at a different wavelength range, or a combination of multiple sensors with filters enabling each sensor to capture different wavelength ranges” (Benny, Page 16);
varying a wave amplitude or a frequency of the spectral marker “In this setting the target spectral signature can be of a given agriculture product that system 100 is trying to identify and further investigate. For example, a spectral signature of a strawberry. The spectral signature of the strawberry is a vector associated with multiple values. Each value in this vector is associated with the spectral variation of reflectance or emittance of a strawberry in the wavelength ranges that can be captured by the multi-spectral sensor 104” (Benny, Page 26) where over a period of time a strawberry will grow from a strawberry plant and the varied frequency will be observed, additionally “The filter wheel is used to rotatably position a filter with a given wavelength range in the imaging path of the sensor to capture an image in that wavelength range. The filter in the imaging path of the sensor at a given time- window and/or interval of time is the active filter at that time window. The wheel is then turned to position the next filter, which has a different wavelength range in the imaging path of the sensor allowing the sensor to capture an image in the different wavelength range. Rotating the filter wheel allows the multi- spectral sensor 104 to capture a series of images in multiple wavelength ranges over a given time-frame.” (Benny, Page 16);
masking at least a portion of the spectral signature of at least one of the spectral markers with a spectral filter disposed on the spectral marker “automatically generate hyperspectral and/or multi-spectral training data-sets are depicted in Fig. 15. At step 1502 a Materials database that stores a reflectance spectrum, a label (materials name), and a representative color for at least one type of material is used together with one or more heuristics tables from step 1504 and a photo-realistic engine of a 3D virtual world (such as: a game engine like the unreal engine, etc.) which includes RGB frames of the virtual world, distance maps within the virtual world, colors maps of scenes and objects within the virtual world and labels (objects) masks (from step 1506) to transplant materials into the virtual world” (Benny, Page 23);
reading each of the spectral markers within the three-dimensional space to determine the spectral signature corresponding to each one of the spectral markers, wherein reading the plurality of spectral markers comprises reading the spectral markers with a spectral device configured to read the pattern of spectral values of each spectral signature, and wherein the spectral device is disposed on a robot payload “In accordance with the presently disclosed subject matter, the real-time multi- spectral system 100 can be optionally enclosed within an observation pod. The observation pod can be used as a stand-alone system and/or can be installed as a special purpose payload on a platform, for example: on a personal platform, on a watch tower as part of a wider border defense system, on a ground vehicle, on an aerial platform, in space, etc. When system 100 is installed in an observation pod on an aerial platform, it can detect potential targets in a given scene in real-time, without the need to download information from the aerial platform to a ground station for human operator analysis. System 100 is capable of keeping observation capabilities on a given observed area by controlling the movement of the multi-spectral sensor's 104 lens and of the lenses of the additional sensors (e.g., the wide daylight sensor 106, the narrow daylight sensor 110, the wide thermal sensor 114, the narrow thermal sensor 118, the narrow SWIR sensor 108, the laser rangefinder 116, the laser pointer 112, etc.) while the platform is in motion.” (Benny, Page 25) where system 100 can be implemented in a pod and placed on a vehicle or robot payload for example,
associating the determined spectral signatures of each of the plurality of spectral markers with a unique identification, wherein the unique identification corresponds to a location within the three-dimensional space “multi- spectral system 100 can be configured to obtain: (A) a machine learning model capable of receiving the image of the scene and identify the existence of at least one object of the automatically identified objects within the scene, each identified object is associated with: (i) a location within the scene, and (ii) a predetermined representative reflective spectral signature, and (B) the image of the scene (block 1002)” (Benny, Page 36) where the spectral signal is identified by its reflective color or location or a combination”;
providing the unique identifications to a robot within the three-dimensional space, wherein the robot aligns itself within the three-dimensional space according to the associated locations “In accordance with the presently disclosed subject matter, the real-time multi- spectral system 100 can be optionally enclosed within an observation pod” (Benny, Page 25) where the observation pod may be attached to another device or location. Additionally, “System 100 is capable of keeping observation capabilities on a given observed area by controlling the movement of the multi-spectral sensor's 104 lens and of the lenses of the additional sensors (e.g., the wide daylight sensor 106, the narrow daylight sensor 110, the wide thermal sensor 114, the narrow thermal sensor 118, the narrow SWIR sensor 108, the laser rangefinder 116, the laser pointer 112, etc.) while the platform is in motion. In many cases the various sensors in pod or pay load, although they might have different FOVs, the center FOV of each sensor is aligned to the same direction, enabling the system to use different sensors for the same target location” (Benny, Page 25)”;
reconstructing a three-dimensional spectral model using the robot, wherein the three-dimensional spectral model comprises a three-dimensional volume including the assigned locations “System 100 can utilize the atmospheric simulator to generate more than one un-calibrated multi-spectral data cube with known target material from a single given calibrated multi-spectral data cube using different atmospheric conditions, thus allowing to easily create a large training data set from a small number of calibrated multi-spectral data cubes which are labeled with known target materials and their location within the scene” (Benny, Page 18) where the model is a data cube and may be labeled with the material and location;
“In some cases, system 100 is not provided with specific target spectral signatures. In these cases, system 100 can determine if one or more anomalies exist in the spectral data cube of the given scene. Such anomalies are detected by system 100 when one or more pixels within the spectral data cube have a spectral signature that is distanced above a threshold distance from the other pixels of the data cube” (Benny, Page 26);
Benny does not teach wherein the pattern of each of the spectral markers comprises:
at least one neutral area;
a second area comprising a spectral value which is higher relative to the first area; and
a third area comprising a spectral value which is lower relative to first area, wherein the first, second, and third areas of each of the spectral markers are arranged in a surface pattern that is unique to each one of the corresponding spectral markers, wherein the second area comprises a spectral value which is higher relative to a spectral value of the at least one neutral area, and wherein the third area comprises a spectral value which is lower relative to the spectral value of the at least one neutral area, wherein the pattern of spectral values that is unique to each one of the corresponding spectral markers is configured to provide a contrast between at least two areas of the spectral marker, wherein the at least one neutral area is comprised of a material configured to provide a contrast with the first, second, or third area;
transmitting a supplemental data signal from at least one of the spectral markers to a user, wherein the supplemental data signal is comprised of at least one of the following: GPS data, humidity, detection or concentration of a gas, pressure, or fluid level;
maintaining a portion of the spectral markers at a respective invariant spectral value, wherein the portion comprises the second and third areas, wherein the spectral markers receive power to maintain the respective invariant spectral value from the three-dimensional space or from an outside or independent power source;
displaying the reconstructed three-dimensional volume, wherein displaying the three-dimensional volume comprises displaying the reconstructed three-dimensional volume on a screen, and
performing a wellsite action in response to the three-dimensional spectral model, wherein performing the wellsite action comprises generating or transmitting a signal that instructs or causes an action to occur, wherein the action comprises a physical action, and wherein the physical action comprises selecting where to drill a wellbore in the subsurface formation, drilling the wellbore, varying a trajectory of the wellbore, varying a weight or torque on a drill bit that is drilling the wellbore, varying a rate or concentration of a fluid being pumped into the wellbore, or a combination thereof.
Brogger teaches at least one neutral area “In various embodiments, the opaque base layers and the spectral taggant layers may be formulated in various ways. For example, opaque base layers useful in any embodiments of the present invention are often formulated to provide a polymer matrix presenting a single, neutral, opaque, color such as an opaque white or grey, but these can be formulated to display one or more other colors or other surface characteristics, if desired” (Brogger, ¶ [0074]);
a second area comprising a spectral value which is higher relative to the first area “FIG. 3 shows an alternative embodiment of a platelet-shaped, one-sided taggant particle 34. In a manner similar to taggant particles 10 of FIGS. 1 and 2, taggant particle 34 also has opposed and parallel major faces 35 and a side 37 that interconnect the major faces 35 around the perimeter of the major faces 35. Using an illustrative manufacturing method described further below, the major faces 35 are generally parallel to each other so that the height of the sides 37 between the major faces 35 is generally uniform. The method may have a tendency to produce taggant particles 34 for which the perimeters 39 defining the major face shapes are somewhat irregular. In contrast to taggant particle 10 of FIGS. 1 and 2, taggant particle 34 deploys taggant material in a stack of multiple spectral taggant layers 38, 44, and 50 provided on opaque base layer 36” (Brogger, ¶ [0066]) where the taggant particles may vary in spectral value (i.e. higher or lower) “Different taggant particles may be used in combination to produce even more complex signatures.” (Brogger, ¶ [0013]); and
a third area comprising a spectral value which is lower relative to first area, wherein the first, second, and third areas of each of the spectral markers are arranged in a surface pattern that is unique to each one of the corresponding spectral markers ““Taggant system 26 generally includes one or more taggants. For purposes of illustration, taggant system 26 includes a combination of taggants 28 and 30 incorporated into the same spectral taggant layer 24. In other modes of practice, each of taggants 28 and 30 could be incorporated into separate spectral taggant layers if desired. Using a combination of two or more spectral taggants 28 and 30 offers many signatures strategies to be implemented” (Brogger, ¶ 0062]) where the taggants may be in a desired pattern with spectral values decreasing further from the center, wherein the second area comprises a spectral value which is higher relative to a spectral value of the at least one neutral area “In contrast to taggant particle 10 of FIGS. 1 and 2, taggant particle 34 deploys taggant material in a stack of multiple spectral taggant layers 38, 44, and 50 provided on opaque base layer 36” (Brogger, ¶ [0066]) where the taggant particles may vary in spectral value (i.e. higher or lower) and have contrast to the opaque base layer, or neutral area “Different taggant particles may be used in combination to produce even more complex signatures.” (Brogger, ¶ [0013]), and wherein the third area comprises a spectral value which is lower relative to the spectral value of the at least one neutral area “Opaque base layer 22 helps to provide a solid background against which the spectral signature or code incorporated into the spectral taggant layer 24 can be produced and read. The solid background helps to allow a better, stronger spectral signal to be read, particularly when the spectral signature is read remotely from a distance. Also important, the solid background helps to produce a consistent spectral output that is less vulnerable to substrate color, translucency, ambient light, and other background noise that could affect reading the output, in comparison tests, spectral signatures of particle embodiments including one or more base color layers would be easily read from a distance using multispectral/hyperspectral imaging techniques” (Brogger, ¶ 0045]) where the multiple taggant layers have a spectral contrast from the opaque base layer, wherein the pattern of spectral values that is unique to each one of the corresponding spectral markers is configured to provide a contrast between at least two areas of the spectral marker “ Fig. 6 shows an alternative embodiment of a platelet-shaped, two-sided taggant particle 90 that incorporates different spectral signatures on each side. To accomplish this, spectral taggant layers … are deployed on one side of opaque base layers 124. In the meantime, spectral taggant layers … are deployed on the other side of opaque base layers 124. The result is that each side of taggant particle 90 encodes a different spectral signature” (Brogger, ¶ [0055]), wherein the at least one neutral area is comprised of a material configured to provide a contrast with the first, second, or third area “The solid background helps to allow a better, stronger spectral signal to be read, particularly when the spectral signature is read remotely from a distance. Also important, the solid background helps to produce a consistent spectral output that is less vulnerable to substrate color, translucency, ambient light, and other background noise that could affect reading the output, in comparison tests, spectral signatures of particle embodiments including one or more base color layers would be easily read from a distance using multispectral/hyperspectral imaging techniques” (Brogger, ¶ [0045]);
Zhang teaches transmitting a supplemental data signal from at least one of the spectral markers to a user, wherein the supplemental data signal is comprised of at least one of the following: GPS data, humidity, detection or concentration of a gas, pressure, or fluid level “Each infrared emission host 111 emits infrared rays to a corresponding infrared reflective assembly 112 to monitor the air mass in the monitored range of the monitoring equipment 11 in real time to obtain the infrared spectral data of the chemical factors in the passing air mass. The pollution factor qualitative module 12 presets the infrared judgment reference data of a plurality of pollution factors, which uses the infrared spectrum values of a plurality of different pollution factors as a reference” (Zhang, ¶ [0041]);
Garsha teaches maintaining a portion of the spectral markers at a respective invariant spectral value, wherein the portion comprises the second and third areas, wherein the spectral markers receive power to maintain the respective invariant spectral value from the three-dimensional space or from an outside or independent power source The power and temperature of the spectrum source 410 may be stabilized and monitored by closed-loop electronics and / or multi-bandpass filters 410a. The multi-bandpass filter 410a has n predetermined passbands and is positioned in front of the spectrum source 410. In an example embodiment of the invention, a spectrum acquisition system 422, eg, a microscope-based light acquisition device, includes or is coupled to a spectrum camera 443. The spectrum acquisition device 442 includes a scanning platform 445” (Garsha, Page 11) where maintaining a stable power of the spectrum source outputs a maintained spectrum;
Profio teaches displaying the reconstructed three-dimensional volume, wherein displaying the three-dimensional volume comprises displaying the reconstructed three-dimensional volume on a screen “… the various embodiments provide a method for displaying time dependent information on a screen” (Profio, ¶ [0037])
Aqrawi teaches performing a wellsite action in response to the three-dimensional spectral model, wherein performing the wellsite action comprises generating or transmitting a signal that instructs or causes an action to occur, wherein the action comprises a physical action, and wherein the physical action comprises selecting where to drill a wellbore in the subsurface formation, drilling the wellbore, varying a trajectory of the wellbore, varying a weight or torque on a drill bit that is drilling the wellbore, varying a rate or concentration of a fluid being pumped into the wellbore, or a combination thereof “The geological model may include one or more geological model features. The one or more geological features may represent one or more physical features of the geological formation. The physical features of the geological formation may be, for example, one or more layers of sandstone, one or more layers of limestone, one or more layers of shale, one or more layers of sand layer, one or more layers of turbidite, and/or one or more fault lines. The presence of any of these physical features and the characteristics of each feature that is present may cause one or more actions of the extraction plan to be generated” (Aqrawi, ¶ [0030]) where the geological model is based on seismic data and the extraction plan is used to perform a wellsite operation such as drilling.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of producing 3-dimensional spectral models taught by Benny with the use of different spectral areas taught by Brogger, using the spectral markers to communicate with a processor taught by Zhang, maintaining a value of the spectral marker taught by Garsha, multiple displays taught by Profio, and the use of drilling taught by Aqrawi to produce a system capable of observing and visualizing spectral markers, which can display to and be interacted with a user and operated to drill in regards to the read spectral information. The motivation to do so would be to produce a system that can bore, controlled by a user, with the goal of extracting or sampling observed spectral models.
Regarding claim 17, Benny in view of Brogger, Zhang, Garsha, Profio, and Aqrawi teach [t]he non-transitory computer-readable medium of Claim 16, wherein displaying the reconstructed three-dimensional volume comprises displaying a first reconstructed three-dimensional volume “… three-dimensional (3D) volume visualization is performed on the 2D stacks of images to form 3D volume visualization models.” (Profio, ¶ [0027]) corresponding to a first spectral range “… system 100 utilizes multi- spectral sensor 104 to capture the given scene in multiple wavelength ranges, as the aerial platform flies over the given scene.” (Benny, Page 25) for a first wavelength range, combined with a second reconstructed three-dimensional volume “The 4D MIP visualizations can then be viewed temporally in time, viewed by spatial location, re-combined with digital addition and subtraction, MIP, weighting, average, minimum MIP, volume rendering, and other mathematical constructs to provide for display, for example, images showing different phases of vascular blood flow and other diagnostic information. It should be noted that 1 to N time based volume data sets may be digitally added or subtracted.” (Profio, ¶ [0027]) where multiple data sets can be added, corresponding to a second spectral range “… system 100 utilizes multi- spectral sensor 104 to capture the given scene in multiple wavelength ranges, as the aerial platform flies over the given scene.” (Benny, Page 25 for a second wavelength range, wherein the first and second reconstructed three-dimensional volumes are displayed on top of one another.
PNG
media_image4.png
544
574
media_image4.png
Greyscale
Profio, Fig. 6, multiple viewports (items 120) stacked in various ways for simultaneous viewing of the 3D model image over time.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of producing 3-dimensional spectral models taught by Benny with the use of multiple displays taught by Profio to produce multiple displays for multiple three-dimensional spectral models. The motivation to do so would be to observe and compare multiple models next to each other.
Regarding claim 18, Benny in view of Brogger, Zhang, Garsha, Profio, and Aqrawi teach [t]he non-transitory computer-readable medium of Claim 16, wherein displaying the reconstructed three-dimensional volume comprises displaying a first reconstructed three-dimensional volume corresponding to a first time period combined with a second reconstructed three-dimensional volume corresponding to a second time period “Thus, as shown in FIG. 5, 2D image reconstruction may be performed during a first processing step 100 wherein a series of 2D image stacks 102 (e.g., sixty-four images in each of thirty-five stacks) are formed. Thereafter, during a second processing step 104, 3D image reconstruction is performed to generate a series of 3D MIP images 106 (e.g., thirty-five MIP images).” (Profio, ¶ [0027]) where a stack, or multiple 3D images are produced, additionally “Thereafter, at 86, three-dimensional (3D) volume visualization is performed on the 2D stacks of images to form 3D volume visualization models. In particular, maximum intensity pixel projection is performed through each stack of 2D images resulting in a 3D MIP volume at each point in time” (Profio, ¶ [0027]) where each point in time has a 3D image, wherein the first and second reconstructed three-dimensional volumes are displayed on top of one another “A display, for example, the 4D MIP viewer 52 as shown more particularly in FIG. 6, may include a plurality of viewports 120 segmenting the display” (Profio, ¶ [0030]) where the 3D models are shown over time in a display that is split for multiple viewports at once.
PNG
media_image4.png
544
574
media_image4.png
Greyscale
Profio, Fig. 6, multiple viewports (items 120) stacked in various ways for simultaneous viewing of the 3D model image over time.
Regarding claim 19, Benny in view of Brogger, Zhang, Garsha, Profio, and Aqrawi teach [t]he non-transitory computer-readable medium of Claim 16, wherein displaying the reconstructed three-dimensional volume comprises displaying the reconstructed three-dimensional volume in one, two, three, or four dimensions “In accordance with one embodiment, a method for displaying medical image data is provided. The method includes performing three-dimensional (3D) volume visualization on a plurality of two-dimensional (2D) images acquired continuously. The method further includes combining the 3D volume visualizations to form a plurality of four-dimensional (4D) volumes representing the 3D volume visualizations over a period of time and displaying at least one 4D volume.” (Profio, ¶ [0006]).
Regarding claim 20, Benny in view of Brogger, Zhang, Garsha, Profio, and Aqrawi teach [t]he non-transitory computer-readable medium of Claim 16, wherein the first area comprises a temperature equal to the three-dimensional space, wherein the second area comprises a temperature that is higher relative to the first area, and wherein the third area comprises a temperature that is lower relative to first area “FIG. 16 shows an alternative embodiment of a taggant particle 610 of the present invention useful to detect vehicles, people, animals, or other mobile subjects have been in a particular area. Particle 610 includes at least one multilayer taggant particle 612 of the present invention that includes at least one spectral taggant layer supported on at least one side of an opaque base layer. In some embodiments, any of the particles from FIGS. 1-6 may be used as taggant particle 612. A light transmissive, tacky adhesive 614 surrounds the core taggant particle(s) 612. An opaque shell 618 encapsulates the taggant particle 612 and adhesive 614. A gap 616 is between the shell 618 and the particle(s) 612” (Brogger, ¶ [0174]) where the spectral taggants may display IR information “A wide variety of one or more different taggants can be used in the spectral taggant layers of FIGS. 1 to 6 as well as other embodiments of spectral taggant. Illustrative taggants include luminescent compounds, IR absorbing compounds, infrared reflecting compounds, ultraviolet absorbing compounds, ultraviolet reflecting compounds, combinations of these, and the like” (Brogger, ¶ [0097]) additionally, taggant system 26 includes a combination of taggants 28 and 30 incorporated into the same spectral taggant layer 24. In other modes of practice, each of taggants 28 and 30 could be incorporated into separate spectral taggant layers if desired. Using a combination of two or more spectral taggants 28 and 30 offers many signatures strategies to be implemented. In some modes of practice, each of taggants 28 and 30 may produce an independent spectral output” (Brogger, ¶ [0062]) where each layer may have a different signature, or heat in the sense of IR.
PNG
media_image5.png
357
628
media_image5.png
Greyscale
Brogger, Figs. 15-17, showing different arrangements of spectral taggants or taggant layers (items 622 & 624), used to identify spectral signatures/presence which reads upon the given pattern.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAIDEN ALEXANDER USSERY whose telephone number is (571)272-1192. The examiner can normally be reached Monday - Friday* 7:30AM - 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571) 272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.A.U./Examiner, Art Unit 2611
/TAMMY PAIGE GODDARD/Supervisory Patent Examiner, Art Unit 2611