DETAILED ACTION
This Office Action is in response to the Applicants' communication filed on April 10, 2026, which amends the independent claims 1 and 10, cancels the dependent claims 5 and 14, and presents arguments, is hereby acknowledged. Claims 1-4, 6-13 and 15-20 are currently pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s arguments filed on April 10, 2026, have been fully considered.
Applicant argues that by this response, the independent claims 1 and 10 are hereby amended to add a new limitation “a multispectral sensor configured to sense different wavelength bands of light reflected from a measurement-target object, wherein the multispectral sensor comprises a plurality of pixels, and each of the plurality of pixels comprises at least four different spectral channels; one or more processors; and
a memory storing instructions that, when executed by the one or more processors, cause the apparatus to: generate N channel images based on signals obtained from a plurality of channels of the multispectral sensor, wherein N is an integer of at least four; select at least one first channel image corresponding to a visible wavelength band from among the N channel images;
generate a RGB reference image based on the at least one first channel image; select a second channel image from remaining channel images of the N channel images, wherein the second channel image corresponds to a reflection wavelength range of a target substance of the measurement-target object; generate object information by analyzing of the measurement-target object by calculating normalized pixel values of the measurement-target object in the second channel image relative to all pixel values of the second channel image; generate an output image by combining the RGB reference image with the object information of the measurement-target object; and display the output image to a user.” in order to overcome the 35 U.S.C. §103 rejection.
Examiner replies that the amended claims have added 7 new limitations, and some of them are disclosed in the prior arts on record, and they are added to clarify the claims. The newly added limitation “different wavelength bands of light” is disclosed by the primary art, Tyan, etc. (US 20220125280 A1) (See Tyan: Fig. 1, and [0069], “A filter 107 is arranged along the optical pathway, and selectively passes the reflected light in a visible light range and a NIR light range toward the image sensor 104. The filter 107 can include a notch filter or a bandpass filter. As a specific example, the filter 107 includes a first bandpass filter to selectively pass visible light or wavelengths and a second bandpass filter to selectively pass NIR light or wavelengths”). Tyan also teaches that “a measurement-target object, wherein the multispectral sensor comprises a plurality of pixels, and each of the plurality of pixels comprises at least four different spectral channels” (See Tyan: Fig. 1, and [0070], “The image sensor 104, which includes circuitry, collects light reflected from the sample 109 in response to the passed first polarization light and second polarization light in the visible and/or MR light range or wavelengths. As further described herein, a plurality of images can be captured at each of the visible light range and the NIR light range, and while the first and second polarizers 105, 106 are at different angles. The image sensor 104 can include a multi-channel sensor, such as a multi-channel camera”; Fig. 17, and [0136], “To process the visible images, the RGB channels of a visible image 1771 are converted into the YCbCr color space, at 1773. At 1774, the CLAHE filter is applied to the luminance component Y, while Cb and Cr channels remain unchanged”; and [0138], “A CLAHE filter is used for both the visible and NIR images to improve the contrast of the images by computing several histograms, each corresponding to a section of the image, and by redistributing lightness values of the images. For example, the images can be divided into sections or tiles of a width and height of the pixels, which can be larger than a size of the features to be observed. As visible light include RGB channels, the visible image frames are converted to the YCbCr space, and the CLAHE filter is applied to the Y channel while the CB and Cr channels remain unchanged. Additionally, the bilateral filter is applied to the Y channel to mitigate weak edge signals from being diminished and the results are combined with the original image by a weighted summation. The NIR frames are processed based on I only and by application of the CLAHE filter and the bilateral filter, with the filter images being combined with the original by a weighted summation. The enhanced output image can have a similar texture as the original”); that “wherein N is an integer of at least four” (See Tyan: Fig. 17, and [0138], “[0138], “A CLAHE filter is used for both the visible and NIR images to improve the contrast of the images by computing several histograms, each corresponding to a section of the image, and by redistributing lightness values of the images. For example, the images can be divided into sections or tiles of a width and height of the pixels, which can be larger than a size of the features to be observed. As visible light include RGB channels, the visible image frames are converted to the YCbCr space, and the CLAHE filter is applied to the Y channel while the CB and Cr channels remain unchanged. Additionally, the bilateral filter is applied to the Y channel to mitigate weak edge signals from being diminished and the results are combined with the original image by a weighted summation. The NIR frames are processed based on I only and by application of the CLAHE filter and the bilateral filter, with the filter images being combined with the original by a weighted summation. The enhanced output image can have a similar texture as the original”)”. Note that 3 RGB channels plus 1 NIR channel is N = 4); and that “RGB reference image” (See Tyan: Fig. 17, and [0136], “To process the visible images, the RGB channels of a visible image 1771 are converted into the YCbCr color space, at 1773. At 1774, the CLAHE filter is applied to the luminance component Y, while Cb and Cr channels remain unchanged”).
Examiner further replies that the amended claims have added 7 new limitations, while 4 of these 7 newly added limitations are disclosed in the primary arts, as mentioned above, the other 3 newly added limitations may overcome the current rejections. However, a newly found art, McQuilkin, etc. (US 20170089761 A1) teaches those newly added limitations, specifically that select a second channel image from remaining channel images of the N channel images, wherein the second channel image corresponds to a reflection wavelength range of a target substance of the measurement-target object (See McQuilkin: Fig. 1, and [0155], “In block 23, the spectral information of the target substance and optionally the background (if desired) is then analyzed via suitable signal processing and spectral analysis techniques to uniquely identify one or more, specific spectral wavelengths that may be used to distinguish the target substance from the background. In preferred modes of practice, at least two specific spectral wavelengths are identified. More preferably, 2 to 500, even more preferably 2 to 100, even more preferably 2 to 50, and even more preferably 2 to 15 specific wavelengths are identified”; [0270], “In another aspect, the present invention relates to a method to analyze substances present in water. As discussed herein, the spectral absorption band of water near 980 nm is narrow. This characteristic permits the identification of water with a filter element sensitive near 980 nm. Adjacent bands, outside of the water absorption band, are available to detect impurities, contaminants or formulations within the water. Examples of such uses include but are not limited to, pollution detection, on-site water analysis, aerial inspection of water quality, beverage processing, wastewater treatment, drinking water treatment, biosecurity at reservoirs, toxicity analyses, presence of organic material, presence of organisms, groundwater analysis and underwater exploration or mining”; and Figs, 25A-E, and [0456], “Referring to FIGS. 25A to 25E, the principles of the present invention are illustrated by an example having two glass vessels partially filled with water that are within the field of view of a spectral imaging system of the present invention. The system is used to detect the presence of water in the two vessels. In each of FIGS. 25A through 25E, the vessel on the left is a round drinking glass. The vessel on the right is a square glass flower vase with a bottom of non-uniform thickness. It is desired to generate a display image that shows the position of the water within each vessel, as well as an indication of the spatial position of objects within the field of view. FIG. 25A is a ‘full spectrum’ image taken with the narrower visible spectrum filter removed (typically included with digital cameras); FIG. 25B is an NIR detection image, sensitive to a narrow bandwidth centered on the absorption peak of water at 980 nm. FIG. 25C is a spectral reference image centered in an area of the spectrum which exhibits little spectral absorption for water at 766 nm. FIG. 25D is a ratio image representing the reference image divided by the NIR image. FIG. 25E is a display image created from the reference image, those regions of the ratio image where the water is present, and a custom colormap to highlight the position of water. A reference spectrum for water is shown in FIG. 2A”. Note that the target object is water here as an example, and specific wavelength signals are selected to detect “water”).
Examiner respectfully further replies that the newly found art, McQuilkin, etc. (US 20170089761 A1)also teaches the remaining newly added limitations, specifically that generate object information by analyzing of the measurement-target object by calculating normalized pixel values of the measurement-target object in the second channel image relative to all pixel values of the second channel image (See McQuilkin: Fig. 1, and [0147], “An illustrative embodiment of a spectral imaging system 10 of the present invention is schematically shown in FIG. 1. Spectral imaging system 10 includes set-up features and operational features. The set-up features, customized for each chosen target substance, involve obtaining the electromagnetic spectrum for the chosen target substance, selecting unique spectral wavelength(s) to identify the substance, selecting corresponding filter elements, and designing a unique target algorithm to reliably identify the target. The operational features include both equipment and processing components. The operational equipment comprises a spectral filter card 106 and a multi-camera array 111 having image capturing elements (such as CCD sensors). The operational processing components may include the following processing steps: image acquisition, image alignment, image normalization for variations in spectral parameters, computation of the presence and/or quantity of the target substance, generation of a colormap, and generation and display of an output image showing the result of the detection procedure. The operational blocks are processing operations that may be executed at or near real time as the multi-camera system is operating and the target substance is within the field of view. While an advantage of the present invention is real-time operation, the present invention also encompasses modes of operation in non-real time mode, batch mode, or other slow or delayed processing operation. Additionally, the simultaneous image acquisition permits freeze-motion images to be easily obtained for moving targets”; [0183], “Once the illumination spectrum is known, the intensity of each acquired image may be normalized in optional block 34 to take the spectral features of the given illumination into account. A variation on the automated identification of the illumination source would include a user interface (i.e., software, switch or button) to manually identify the type of illumination as shown by optional block 27”; [0235], “For example, when the principles of the present invention are used to detect water as described herein and depending upon other materials in the background, a simple ratio between the spectra sampled at two different wavelengths would be quite effective to detect water in a field of view of a spectral imaging system. If spectral information for a particular image pixel demonstrates such a ratio, the system may accurately conclude that water is present in the captured image at that pixel location”; and [0267], “In one aspect, the present invention relates to a method to derive information about water within a target field of view. If the target substance is a volume of water, the selected wavelengths might be one wavelength in which water is absorbed, such as 980 nm, and one wavelength in which water exhibits little absorption, such as 766 nm. Filter elements would contain bandpass filters for each of the selected wavelengths. After image alignment, a simple ratio of image intensities for corresponding pixels would yield an indication of the presence of water”. Note that the image pixel values are normalized, and water as an example, a simple ratio of the pixel values is used to detect the presence of water in the field of views); and that generate an output image by combining the RGB reference image with the object information of the measurement-target object (See McQuilkin: Fig. 1, and [0188], “FIG. 1 shows an exemplary technique for generating a “presence” output in the form of a displayed image in which locations of the target substance in the displayed image are colored substantially differently from other portions of the image. According to this technique, the acquired images can be converted in block 37 into an output image 122 of the target surface 109 showing the locations 120 of the target substance on the image 122. The matrix of pixels making up image 122 may be enhanced to more clearly show the locations of the target substance by color-coding the image. If this is desired, a suitable color map may be selected in optional block 36 and then applied in block 37 to output the color-coded image 122. As a result, the locations 120 of the target substance are shown as areas of a different color relative to the other portions of image 122. In many embodiments, the color used to represent the locations of the target substance can be much brighter than the other colors in image 122 to make it easy to see the locations 120 in image 122”; [0303], “In another embodiment, the present invention relates to a method to detect a target substance that is a specific biofilm on a surface. A representative biofilm include films generated by microorganisms such as E. Coli or Salmonella. In this application a characteristic spectrum of the specific biofilm is obtained by conventional hyperspectral imaging methods or other spectral means. Then specific selected wavelengths are determined that identify the desired spectra. Filter elements passing the selected wavelengths are incorporated into the elements of the spectral filter array. A target algorithm is established to reliably detect the chosen target substance. The present invention acquires and processes a number of spectral images as described herein. Finally, custom color displays may be generated to highlight the presence of a given biofilm on a surface within the field of view of the system”; and Figs. 25A-E, and [0460], “In another embodiment, the present invention relates to a method to detect a target substance that is a specific biofilm on a surface. A representative biofilm include films generated by microorganisms such as E. Coli or Salmonella. In this application a characteristic spectrum of the specific biofilm is obtained by conventional hyperspectral imaging methods or other spectral means. Then specific selected wavelengths are determined that identify the desired spectra. Filter elements passing the selected wavelengths are incorporated into the elements of the spectral filter array. A target algorithm is established to reliably detect the chosen target substance. The present invention acquires and processes a number of spectral images as described herein. Finally, custom color displays may be generated to highlight the presence of a given biofilm on a surface within the field of view of the system”. Note that the substance (water) is detected, and color images of the target object water) are generated for the user). The remaining arguments of the applicant are mooted in view of the newly found art.
Examiner respectfully further replies that the Applicant's arguments have been fully considered and a new ground of rejections have been made. Accordingly, new grounds of rejection are set forth below. Since the new grounds of rejection are necessitated by Applicant's amendments to the claims, the present action is made final.
Claim Objections
Claim 11 is objected to because of the following informalities: “the reference image” may be “the RGB reference image”. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 6-7, 10-13, 15-17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Tyan, etc. (US 20220125280 A1) in view of Xiao, etc. (US 20240185389 A1), further in view of McQuilkin, etc. (US 20170089761 A1).
Regarding claim 1, Tyan teaches that an apparatus for acquiring images (See Tyan: Fig. 1, and [0065], "Turning now to the figures, FIG. 1 illustrates an example of an apparatus for imaging, in accordance with various embodiments. The apparatus can be used for imaging of soft tissue using cross-angled polarized light, as further described herein"), the apparatus comprising:
a multispectral sensor configured to sense different wavelength bands of light (See Tyan: Fig. 1, and [0069], “A filter 107 is arranged along the optical pathway, and selectively passes the reflected light in a visible light range and a NIR light range toward the image sensor 104. The filter 107 can include a notch filter or a bandpass filter. As a specific example, the filter 107 includes a first bandpass filter to selectively pass visible light or wavelengths and a second bandpass filter to selectively pass NIR light or wavelengths”) reflected from a measurement-target object (See Tyan: Fig. 1, and [0070], "The image sensor 104, which includes circuitry, collects light reflected from the sample 109 in response to the passed first polarization light and second polarization light in the visible and/or MR light range or wavelengths. As further described herein, a plurality of images can be captured at each of the visible light range and the NIR light range, and while the first and second polarizers 105, 106 are at different angles. The image sensor 104 can include a multi-channel sensor, such as a multi-channel camera". Note that the multi-channel image sensor is mapped to the multispectral image sensor), wherein the multispectral sensor comprises a plurality of pixels, and each of the plurality of pixels comprises at least four different spectral channels (See Tyan: Fig. 1, and [0070], “The image sensor 104, which includes circuitry, collects light reflected from the sample 109 in response to the passed first polarization light and second polarization light in the visible and/or MR light range or wavelengths. As further described herein, a plurality of images can be captured at each of the visible light range and the NIR light range, and while the first and second polarizers 105, 106 are at different angles. The image sensor 104 can include a multi-channel sensor, such as a multi-channel camera”; Fig. 17, and [0136], “To process the visible images, the RGB channels of a visible image 1771 are converted into the YCbCr color space, at 1773. At 1774, the CLAHE filter is applied to the luminance component Y, while Cb and Cr channels remain unchanged”; and [0138], “A CLAHE filter is used for both the visible and NIR images to improve the contrast of the images by computing several histograms, each corresponding to a section of the image, and by redistributing lightness values of the images. For example, the images can be divided into sections or tiles of a width and height of the pixels, which can be larger than a size of the features to be observed. As visible light include RGB channels, the visible image frames are converted to the YCbCr space, and the CLAHE filter is applied to the Y channel while the CB and Cr channels remain unchanged. Additionally, the bilateral filter is applied to the Y channel to mitigate weak edge signals from being diminished and the results are combined with the original image by a weighted summation. The NIR frames are processed based on I only and by application of the CLAHE filter and the bilateral filter, with the filter images being combined with the original by a weighted summation. The enhanced output image can have a similar texture as the original”);
one or more processors (See Tyan: Fig. 1, and [0076], "As may be appreciated, the processing circuitry 110 (sometimes referred to as "a processor") can be implemented as a multi-core processor or a processor circuit implemented as a set of processor circuits integrated as a chip set. The processing circuitry 110 can thereby include a single, or multiple computer circuits including memory circuitry for storing and accessing the firmware or program code to be accessed or executed as instructions to perform the related operation(s)"); and
a memory storing instructions that, when executed by the one or more processors, cause the apparatus (See Tyan: Fig. 6, and [0095], "The computing device has processing circuitry, such as the illustrated processor 640, and computer readable medium 642 storing a set of instructions 644, 646, 648, 650. The computer readable medium 642 can, for example, include read-only memory (ROM), random-access memory (RAM), electrically erasable programmable read-only memory (EEPROM), Flash memory, a solid state drive, and/or discrete data register sets. The computing device illustrated by FIG. 6 can form part of the imaging device having the image sensor, such as the processor 640 including part of the control circuitry illustrated by FIG. 1") to:
generate N channel images based on signals obtained from a plurality of channels of the multispectral sensor, wherein N is an integer of at least four (See Tyan: Figs. 3A-B, and [0081], "Four images, R, G, B, and NIR, are extracted from the corresponding channels in the raw image. The polarized signal can be extracted from the two images captured with and without polarization illumination, such as two NIR images captured with and without polarization illumination. Compared to other systems addressing the special lighting issue in an operation room, the imaging apparatus, as described herein, can be very sim pie and uses a compact way of providing high sensitivity in detecting soft tissue signal(s). The imaging system can capture color reflectance and NIR images in real time, with the color reflectance image and NIR image being well-aligned because both are captured at the same time". Note that 4 channel images are mapped to N-channel images with N = 4 >0);
select at least one first channel image corresponding to a visible wavelength band from among the N channel images (See Tyan: Fig. 1, and [0069], "TA filter 107 is arranged along the optical pathway, and selectively passes the reflected light in a visible light range and a NIR light range toward the image sensor 104. The filter 107 can include a notch filter or a bandpass filter. As a specific example, the filter 107 includes a first bandpass filter to selectively pass visible light or wavelengths"; and [0098], "Once the plurality of NIR image frames are captured, the computing device adjusts the first polarizer and second polarizer to the first polarization angle and the second polarization angle, and causes the filter to selectively pass the visible light and to generate a visible light image frame. The computing device repeats the adjustment of the polarizers and captures the plurality of visible image frames using each of the different polarization angles of the set". The visible lights are allowed to pass and generate the visible image, and this is mapped to "select at least one first channel image corresponding to a visible wavelength band from among the N channel images");
generate a RGB reference image (See Tyan: Fig. 17, and [0136], “To process the visible images, the RGB channels of a visible image 1771 are converted into the YCbCr color space, at 1773. At 1774, the CLAHE filter is applied to the luminance component Y, while Cb and Cr channels remain unchanged”) based on the at least one first channel image (See Tyan: Fig. 1, and [0074], "In specific embodiments, the control circuitry 108 collects the image data by capturing first image data using collimated incident light as generated by the first and second polarizer 105, 106 and capturing second image data using non-polarized light from the light source 103. The non-polarized light can be used to capture an image that is used as a reference, which is compared to the other images captured using polarized light. Additionally, the reference (e.g., a reference image captured with non-polarized light) can be used as a baseline for fusing with the other images to form an optimal and/or enhanced image. In some specific embodiments, the captured first and second image data includes still image frames and/or video of the sample 109"; and Figs. 18A-B, and [0139], "FIGS. 18A-18B illustrate an example of an image alignment module, in accordance with various embodiments. The image alignment module is used to align images when image sensor (e.g., camera) motion is introduced during image acquisition. Although the image sensor illustrated herein is shown as being immobile, image sensor motion can occur when the bandpass filters are switched to acquire multi-spectral image, among other reasons. This is a design issue that can be avoided, but may be inevitable for practical clinical use where the image sensor is in motion, such as in constant motion. To estimate such a camera motion, it can be assumed that the relationship between two images (e.g., reference image and target image) is a rigid transformation. The image alignment module can be used or be based on ORB approach to match the corresponding points with each other, and which is used to align image frames between one another and combine a plurality of NIR frames into a single NIR frame and a plurality of visible image frames into a single visible image frame". Note that the RGB reference images are visible images, and the invisible NIR images are the target images, thus, the visible image is mapped to the reference image);
select a second channel image from remaining channel images of the N channel images (See Tyan: Fig. 1, and [0069], "A filter 107 is arranged along the optical pathway, and selectively passes the reflected light in a visible light range and a NIR light range toward the image sensor 104. The filter 107 can include a notch filter or a bandpass filter. As a specific example, the filter 107 includes a first bandpass filter to selectively pass visible light or wavelengths and a second bandpass filter to selectively pass NIR light or wavelengths"; and [0072], "In various embodiments, a plurality of NIR images and a plurality of visible light images, using different angled polarized light, can be collected. The control circuitry 108 can sequence through the different polarization angles for NIR images followed by visible light images, and in other embodiments, can collect an NIR image and a visible light image (prior to rotating to different polarized angles) and sequences through the different polarization angles". Note that a plurality of NIR images is mapped to the second channel images), wherein the second channel image corresponds to a reflection wavelength range of a target substance of the measurement-target object;
generate object information by analyzing of the measurement-target object by calculating normalized pixel values of the measurement-target object in the second channel image relative to all pixel values of the second channel image;
generate an output (See Tyan: Fig. 9, and [0143], "FIG. 19 illustrates an example of an image fusion module, in accordance with various embodiments. Combining near-infrared information with visible images can further enhance the contrast and details and produce more vivid colors. This combination can be achieved using fusion algorithm, based on pyramid transform with a pattern selective approach. FIG. 19 shows the general framework with two source images for simple illustration") image by combining the RGB reference image with the object information of the measurement-target object; and
display the output image to a user (See Tyan: Fig. 26, and [0165], "FIG. 26 illustrates example polarized images with fusion of enhancement and detection for features. More specifically, FIG. 26 illustrates fusion results from both visible and NIR for image contrast enhancement and object segmentation are demonstration, where neither the visible nor the NIR image alone provides complete information but the combined image does. This combined result can be rendered to surgeons as the final output from the system. Image 2671 is an enhanced polarized VIS image, image 2673 is an enhanced polarized NIR image, and image 2675 is fused image with nerve enhanced. Image 2672 is an enhanced polarized VIS image with nerve detected, image 2674 is an enhanced polarized NIR image with nerve detected, and image 2676 is fused image with nerve detected". Note that the display of the fused images 2675 and 2676 is mapped to display the output image to a user).
However, Tyan fails to explicitly disclose that wherein the second channel image corresponds to a reflection wavelength range of a target substance of the measurement-target object; generate object information by analyzing of the measurement-target object by calculating normalized pixel values of the measurement-target object in the second channel image relative to all pixel values of the second channel image; generate an output image by combining the RGB reference image with the object information of the measurement-target object.
However, Xiao teaches that generate object information by analyzing of the measurement-target object (See Xiao: Fig. 1, and [0017], "It should be understood that in this embodiment of this application, due to different reflectance of near-infrared light for different types of objects, different objects in the third image include different detail information; therefore, an image region in which a photographed object of a target type (for example, green plants or distant mountains) is located may be obtained from the third image based on the third mask (for example, a semantic segmentation mask); and fusion processing is then performed on the image region and the fourth image, to improve local detail information of the fused image"; and [0296], "It should be understood that for part of scenery (for example, green plants or distant mountains), the NIR image has more detail information. For example, because the green scene has a higher reflectance for near-infrared light, the NIR image has more detail information of the green plants than the RGB image. For example, near-infrared light has longer wavelength than visible light, and the near-infrared light features a stronger diffraction capability. In addition, light with longer wavelength has stronger penetration, and therefore a captured image has a stronger sense of transparency. Compared with the RGB image, the NIR image includes more detail information (for example, texture information of distant mountains) for distant scenery (for example, distant mountains). A local region (for example, an image region with more detail information) may be selected from the NIR image and fused with the RGB image, so as to enhance local detail information of the fused image and improve the image quality of the fused image". Note that the third image is NIR image, which is equivalent to the second image of the instant application, and the NIR image gives out more information of the objects for the fused image to enhance the final image, thus this is mapped to "generate object information by analyzing the second channel image") by calculating normalized pixel values of the measurement-target object in the second channel image relative to all pixel values of the second channel image.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Tyan to have generate object information by analyzing of the measurement-target object as taught by Xiao in order to enhance the detailed information in the image (See Xiao: Fig. 1, and [0175], "Through fusion processing on the third image and the fourth image, multi-spectrum information fusion on near-infrared light image information and visible light image information can be implemented to make the fused image include more detail information. In addition, fusion processing is performed on the third image and the fourth image based on the at least two masks, and image enhancement can be performed from at least two aspects such as sharpness, ghosting removal, or local details. In this way, image enhancement is implemented for the second image obtained by the second camera module (for example, the main-camera module) to enhance the detail information in the image and improve the image quality"). Tyan teaches a method and system that may generate an enhanced image by fusing the visible light images with the NIR image captured by the multispectral sensors; while Xiao teaches a system and method that may analyze the NIR image to get the object information such as location, mask, textures, shape, etc. and fuse them into the RGB (visible) image to obtain enhanced fusion images. Therefore, it is obvious to one of ordinary skill in the art to modify Tyan by Xiao to generate objection information and fuse them into the RGB image to obtain the enhanced fusion image. The motivation to modify Tyan by Xiao is "Use of known technique to improve similar devices (methods, or products) in the same way".
However, Tyan, modified by Xiao. fails to explicitly disclose that wherein the second channel image corresponds to a reflection wavelength range of a target substance of the measurement-target object; by calculating normalized pixel values of the measurement-target object in the second channel image relative to all pixel values of the second channel image; generate an output image by combining the RGB reference image with the object information of the measurement-target object.
However, McQuilkin teaches that wherein the second channel image corresponds to a reflection wavelength range of a target substance of the measurement-target object (See McQuilkin: Fig. 1, and [0155], “In block 23, the spectral information of the target substance and optionally the background (if desired) is then analyzed via suitable signal processing and spectral analysis techniques to uniquely identify one or more, specific spectral wavelengths that may be used to distinguish the target substance from the background. In preferred modes of practice, at least two specific spectral wavelengths are identified. More preferably, 2 to 500, even more preferably 2 to 100, even more preferably 2 to 50, and even more preferably 2 to 15 specific wavelengths are identified”; [0270], “In another aspect, the present invention relates to a method to analyze substances present in water. As discussed herein, the spectral absorption band of water near 980 nm is narrow. This characteristic permits the identification of water with a filter element sensitive near 980 nm. Adjacent bands, outside of the water absorption band, are available to detect impurities, contaminants or formulations within the water. Examples of such uses include but are not limited to, pollution detection, on-site water analysis, aerial inspection of water quality, beverage processing, wastewater treatment, drinking water treatment, biosecurity at reservoirs, toxicity analyses, presence of organic material, presence of organisms, groundwater analysis and underwater exploration or mining”; and Figs, 25A-E, and [0456], “Referring to FIGS. 25A to 25E, the principles of the present invention are illustrated by an example having two glass vessels partially filled with water that are within the field of view of a spectral imaging system of the present invention. The system is used to detect the presence of water in the two vessels. In each of FIGS. 25A through 25E, the vessel on the left is a round drinking glass. The vessel on the right is a square glass flower vase with a bottom of non-uniform thickness. It is desired to generate a display image that shows the position of the water within each vessel, as well as an indication of the spatial position of objects within the field of view. FIG. 25A is a ‘full spectrum’ image taken with the narrower visible spectrum filter removed (typically included with digital cameras); FIG. 25B is an NIR detection image, sensitive to a narrow bandwidth centered on the absorption peak of water at 980 nm. FIG. 25C is a spectral reference image centered in an area of the spectrum which exhibits little spectral absorption for water at 766 nm. FIG. 25D is a ratio image representing the reference image divided by the NIR image. FIG. 25E is a display image created from the reference image, those regions of the ratio image where the water is present, and a custom colormap to highlight the position of water. A reference spectrum for water is shown in FIG. 2A”. Note that the target object is water here as an example, and specific wavelength signals are selected to detect “water”);
by calculating normalized pixel values of the measurement-target object in the second channel image relative to all pixel values of the second channel image (See McQuilkin: Fig. 1, and [0147], “An illustrative embodiment of a spectral imaging system 10 of the present invention is schematically shown in FIG. 1. Spectral imaging system 10 includes set-up features and operational features. The set-up features, customized for each chosen target substance, involve obtaining the electromagnetic spectrum for the chosen target substance, selecting unique spectral wavelength(s) to identify the substance, selecting corresponding filter elements, and designing a unique target algorithm to reliably identify the target. The operational features include both equipment and processing components. The operational equipment comprises a spectral filter card 106 and a multi-camera array 111 having image capturing elements (such as CCD sensors). The operational processing components may include the following processing steps: image acquisition, image alignment, image normalization for variations in spectral parameters, computation of the presence and/or quantity of the target substance, generation of a colormap, and generation and display of an output image showing the result of the detection procedure. The operational blocks are processing operations that may be executed at or near real time as the multi-camera system is operating and the target substance is within the field of view. While an advantage of the present invention is real-time operation, the present invention also encompasses modes of operation in non-real time mode, batch mode, or other slow or delayed processing operation. Additionally, the simultaneous image acquisition permits freeze-motion images to be easily obtained for moving targets”; [0183], “Once the illumination spectrum is known, the intensity of each acquired image may be normalized in optional block 34 to take the spectral features of the given illumination into account. A variation on the automated identification of the illumination source would include a user interface (i.e., software, switch or button) to manually identify the type of illumination as shown by optional block 27”; [0235], “For example, when the principles of the present invention are used to detect water as described herein and depending upon other materials in the background, a simple ratio between the spectra sampled at two different wavelengths would be quite effective to detect water in a field of view of a spectral imaging system. If spectral information for a particular image pixel demonstrates such a ratio, the system may accurately conclude that water is present in the captured image at that pixel location”; and [0267], “In one aspect, the present invention relates to a method to derive information about water within a target field of view. If the target substance is a volume of water, the selected wavelengths might be one wavelength in which water is absorbed, such as 980 nm, and one wavelength in which water exhibits little absorption, such as 766 nm. Filter elements would contain bandpass filters for each of the selected wavelengths. After image alignment, a simple ratio of image intensities for corresponding pixels would yield an indication of the presence of water”. Note that the image pixel values are normalized, and water as an example, a simple ratio of the pixel values is used to detect the presence of water in the field of views); and
generate an output image by combining the RGB reference image with the object information of the measurement-target object (See McQuilkin: Fig. 1, and [0188], “FIG. 1 shows an exemplary technique for generating a “presence” output in the form of a displayed image in which locations of the target substance in the displayed image are colored substantially differently from other portions of the image. According to this technique, the acquired images can be converted in block 37 into an output image 122 of the target surface 109 showing the locations 120 of the target substance on the image 122. The matrix of pixels making up image 122 may be enhanced to more clearly show the locations of the target substance by color-coding the image. If this is desired, a suitable color map may be selected in optional block 36 and then applied in block 37 to output the color-coded image 122. As a result, the locations 120 of the target substance are shown as areas of a different color relative to the other portions of image 122. In many embodiments, the color used to represent the locations of the target substance can be much brighter than the other colors in image 122 to make it easy to see the locations 120 in image 122”; [0303], “In another embodiment, the present invention relates to a method to detect a target substance that is a specific biofilm on a surface. A representative biofilm include films generated by microorganisms such as E. Coli or Salmonella. In this application a characteristic spectrum of the specific biofilm is obtained by conventional hyperspectral imaging methods or other spectral means. Then specific selected wavelengths are determined that identify the desired spectra. Filter elements passing the selected wavelengths are incorporated into the elements of the spectral filter array. A target algorithm is established to reliably detect the chosen target substance. The present invention acquires and processes a number of spectral images as described herein. Finally, custom color displays may be generated to highlight the presence of a given biofilm on a surface within the field of view of the system”; and Figs. 25A-E, and [0460], “In another embodiment, the present invention relates to a method to detect a target substance that is a specific biofilm on a surface. A representative biofilm include films generated by microorganisms such as E. Coli or Salmonella. In this application a characteristic spectrum of the specific biofilm is obtained by conventional hyperspectral imaging methods or other spectral means. Then specific selected wavelengths are determined that identify the desired spectra. Filter elements passing the selected wavelengths are incorporated into the elements of the spectral filter array. A target algorithm is established to reliably detect the chosen target substance. The present invention acquires and processes a number of spectral images as described herein. Finally, custom color displays may be generated to highlight the presence of a given biofilm on a surface within the field of view of the system”. Note that the substance (water) is detected, and color images of the target object water) are generated for the user).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Tyan to have wherein the second channel image corresponds to a reflection wavelength range of a target substance of the measurement-target object; by calculating normalized pixel values of the measurement-target object in the second channel image relative to all pixel values of the second channel image; generate an output image by combining the RGB reference image with the object information of the measurement-target object as taught by McQuilkin in order to enable remote and non-invasive detection of presence and location of one or more target substances in scene (See McQuilkin: Fig. 1, and [0024], "T In many modes of practice, the present invention satisfies the desire to be able to detect a target substance remotely and noninvasively. This avoids the time and cost associated with a laboratory analysis. Using an appropriate spectral filter array according to the principles of the present invention, the system may be used to detect a target substance that is solid, liquid, gas or plasma. The target substance may be a pure chemical substance, a compound of chemical substances, or a mixture of chemical substances. The present invention can be used to detect any kind of target substance that has spectral characteristics that can be viewed within at least portions of the fields of view of the image capture array"). Tyan teaches a method and system that may generate an enhanced image by fusing the visible light images with the NIR image captured by the multispectral sensors; while McQuilkin teaches a system and method that may noninvasively and remotely detect the presence, location, and/or quantity of a target substance in a scene via a spectral imaging system comprising a spectral filter array and image capture array by selecting wavelengths for target object detections and combining the detect target object information with the RGB reference images captured and filtered from the multispectral sensor data to generate output images for the users. Therefore, it is obvious to one of ordinary skill in the art to modify Tyan by McQuilkin to select wavelengths for target object detection and combine the detected substance information with the RGN reference images to generate output images for users. The motivation to modify Tyan by McQuilkin is "Use of known technique to improve similar devices (methods, or products) in the same way".
Regarding claim 2, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 1 as outlined above. Further, Tyan teaches that the apparatus of claim 1, wherein the at least one first channel image comprises a channel-R image, a channel-G image, and a channel-B image (See Tyan: Figs. 3A-B, and [0082], "FIG. 3A, more specifically, illustrates an example color filter array 316. As shown by the insert, the color filter array 316 includes an array of four channels 315, 317, 319, 321. The four channels include NIR 315, Red (R) 317, Blue (B) 319, and Green (G) 321. FIG. 3B is a graph 318 showing an example spectral sensitivity for the notch filters which include NIR 315A, R 317A, B 319A, and G 321A. As shown in FIG. 3B, there is crosstalk between the R, G, B, and NIR channels. The color image channels, e.g., R, Ga nd B, are sensitive to the NIR component. The NIR channel may also not be ideal and is sensitive to visible light, an image enhancement technique together with calibration can be used to extract color and NIR imagery from raw images more accurately").
Regarding claim 3, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 1 as outlined above. Further, Xiao teaches that the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
calculate the object information based on pixel values of at least one region of interest of the second channel image (See Xiao: Fig. 1, and [0253], "For example, the target mask and the third image are multiplied pixel by pixel, and the local image region in the third image may be determined based on pixel values of different regions in the target mask. For example, a local image region in the third image corresponding to an image region with a pixel of 1 in the target mask may be obtained, and the fused image may be obtained through fusion processing on the fourth image and the local image region in the third image". Note that the third image of Xiao is the NIR image, mapped to the second channel image, and the pixel values are used to determine the region/object mask information, which is mapped to the object information).
Regarding claim 4, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 3 as outlined above. Further, Xiao teaches that the apparatus of claim 3, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
calculate the object information based on an average value of the pixel values of the at least one region of interest (See Xiao: Fig. 1, and [0131], "Exposure is an exposure time, Aperture is an aperture size, lso is light sensitivity, and Luma is an average value of Yin an XYZ space of an image". Note that the average Luma Y in XYZ space of the image is mapped to the average value of the pixel values of the at least one region of interest).
Regarding claim 6, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 1 as outlined above. Further, Xiao teaches that the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
extract an edge image from the second channel image (See Xiao: Fig. 12, and [0435], "Optionally, the edge difference image may be obtained based on a union of the high-frequency information of the NIR image and the high-frequency information of the RGB image". Note that the edge difference image is mapped to the edge image); and
add the edge image to the object information (See Xiao: Fig. 12, and [0446], "In this embodiment of this application, the high-frequency information in the NIR image and the RGB image is obtained, so as to obtain the edge difference image. The edge difference image includes all the high-frequency information in the NIR image and the RGB image, and the edge difference image may be used for obtaining the local image region from both the NIR image and the RGB image. Registration processing is then performed on the two local regions, for example, registration processing is performed on the local region in the NIR image by using the local image region in the RGB image as a benchmark, so as to implement registration between the high-frequency information in the NIR image and the high-frequency information in the RGB image, thereby avoiding ghosting in the fused image to some extent". Note that the edge information registration is mapped to add the edge image to the object information).
Regarding claim 7, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 6 as outlined above. Further, Xiao teaches that the apparatus of claim 6, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
combine the edge image with the reference image to generate the output image, wherein the edge image and the reference image are combined by performing at least one of:
adding the edge image to the reference image (See Xiao: Fig. 8, and [0375], "In this embodiment of this application, the electronic device may include the first camera module and the second camera module. The first camera module is a near-infrared camera module or an infrared camera module. The first camera module may capture the N IR Raw image and the second camera module may capture the RGB Raw image. Image processing is performed on the NIR Raw image and the RGB Raw image to obtain the NIR image and the RGB image, respectively. Fusion processing is performed on the NIR image and the RGB image based on at least two masks to obtain a fused image. Because the NIR image is a near-infrared image or an infrared image, the NIR image may include information that cannot be obtained from the RGB image. Through fusion processing on the NIR image and the RGB image, multi-spectrum information fusion on near-infrared light image information and visible light image information can be implemented to make the fused image include more detail information. In addition, fusion processing on the NIR image and the RGB image is performed based on the at least two masks, and image enhancement can be performed from at least two aspects such as sharpness, ghosting removal, or local details. In this way, image enhancement is implemented forth RGB image obtained by the second camera module (for example, the main-camera module) to enhance the detail information in the image and improve the image quality". Note that the edge different image is registered locally, and it is fused into the RGB image (reference image) to generate the final enhanced image, this is mapped to adding the edge image to the reference image); and
subtracting the edge image from the reference image (See Xiao: Fig. 8, and [0375], "In this embodiment of this application, the electronic device may include the first camera module and the second camera module. The first camera module is a near-infrared camera module or an infrared camera module. The first camera module may capture the NIR Raw image and the second camera module may capture the RGB Raw image. Image processing is performed on the NIR Raw image and the RGB Raw image to obtain the NIR image and the RGB image, respectively. Fusion processing is performed on the NIR image and the RGB image based on at least two masks to obtain a fused image. Because the NIR image is a near-infrared image or an infrared image, the NIR image may include information that cannot be obtained from the RGB image. Through fusion processing on the NIR image and the RGB image, multi-spectrum information fusion on near-infrared light image information and visible light image information can be implemented to make the fused image include more detail information. In addition, fusion processing on the NIR image and the RG B image is performed based on the at least two masks, and image enhancement can be performed from at least two aspects such as sharpness, ghosting removal, or local details. In this way, image enhancement is implemented forth RGB image obtained by the second camera module (for example, the main-camera module) to enhance the detail information in the image and improve the image quality". Note that the edge different image is registered locally, and it is fused into the RGB image (reference image) to generate the final enhanced image using two masks, when the mask from the NIR image is used, i.e., the edge image from the NIR image is fused into the RGB reference image, the edge image from the RGB reference image is replaced by the NIR edge image, which is mapped to subtracting the edge image from the reference image).
Regarding claim 10, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 1 as outlined above. Further, Tyan, Xiao, and McQuilkin teach that an electronic device for acquiring images, the electronic device (See Tyan: Fig. 1, and [0065], "Turning now to the figures, FIG. 1 illustrates an example of an apparatus for imaging, in accordance with various embodiments. The apparatus can be used for imaging of soft tissue using cross-angled polarized light, as further described herein") comprising:
a multispectral sensor configured to sense different wavelength bands of light (See Tyan: Fig. 1, and [0069], “A filter 107 is arranged along the optical pathway, and selectively passes the reflected light in a visible light range and a NIR light range toward the image sensor 104. The filter 107 can include a notch filter or a bandpass filter. As a specific example, the filter 107 includes a first bandpass filter to selectively pass visible light or wavelengths and a second bandpass filter to selectively pass NIR light or wavelengths”) reflected from a measurement-target object (See Tyan: Fig. 1, and [0070], "The image sensor 104, which includes circuitry, collects light reflected from the sample 109 in response to the passed first polarization light and second polarization light in the visible and/or MR light range or wavelengths. As further described herein, a plurality of images can be captured at each of the visible light range and the NIR light range, and while the first and second polarizers 105, 106 are at different angles. The image sensor 104 can include a multi-channel sensor, such as a multi-channel camera". Note that the multi-channel image sensor is mapped to the multispectral image sensor), wherein the multispectral sensor comprises a plurality of pixels, and each of the plurality of pixels comprises at least four different spectral channels (See Tyan: Fig. 1, and [0070], “The image sensor 104, which includes circuitry, collects light reflected from the sample 109 in response to the passed first polarization light and second polarization light in the visible and/or MR light range or wavelengths. As further described herein, a plurality of images can be captured at each of the visible light range and the NIR light range, and while the first and second polarizers 105, 106 are at different angles. The image sensor 104 can include a multi-channel sensor, such as a multi-channel camera”; Fig. 17, and [0136], “To process the visible images, the RGB channels of a visible image 1771 are converted into the YCbCr color space, at 1773. At 1774, the CLAHE filter is applied to the luminance component Y, while Cb and Cr channels remain unchanged”; and [0138], “A CLAHE filter is used for both the visible and NIR images to improve the contrast of the images by computing several histograms, each corresponding to a section of the image, and by redistributing lightness values of the images. For example, the images can be divided into sections or tiles of a width and height of the pixels, which can be larger than a size of the features to be observed. As visible light include RGB channels, the visible image frames are converted to the YCbCr space, and the CLAHE filter is applied to the Y channel while the CB and Cr channels remain unchanged. Additionally, the bilateral filter is applied to the Y channel to mitigate weak edge signals from being diminished and the results are combined with the original image by a weighted summation. The NIR frames are processed based on I only and by application of the CLAHE filter and the bilateral filter, with the filter images being combined with the original by a weighted summation. The enhanced output image can have a similar texture as the original”);
an input unit configured to receive a user input (See Xiao: Fig. 17, and [0496], "The processor 1101 may be configured to control the electronic device 1100, execute a software program, and process data of the software program. The electronic device 1100 may further include a communications unit 1105, to implement signal input (reception) and output (transmission)". Note that the communications unit 1105, to implement signal input (reception) is mapped to the input unit);
a display (See Xiao: Fig. 1, and [0151], "The display 194 may be configured to display images or videos");
a processor configured to generate and display an output image on the display based on a multispectral signal received from the multispectral sensor and an input signal received from the input unit (See Tyan: Fig. 6, and [0096], "The computing device can be used to capture image data of a sample, the image data including and/or being indicative of a plurality of polarized NIR image frames and a plurality of polarized visible light image frames of the sample collected using a plurality of different polarization angles of illumination light and imaging light. For example, at 644, the computing device adjusts the first polarizer and second polarizer to a first polarization angle and a second polarization angle, where the first and second polarization angles are crossed with each other (e.g., orthogonal or slant). The adjustment can include causing or controlling a motor to physically rotate or otherwise change the angle of each polarizer or causing different electrical fields to be applied to the respective polarizers. At 646, the computing device generates an NIR image frame of the sample, and at 648, generates a visible light image frame of the sample while the polarizers are at the first and second polarization angles. In some embodiments, the image sensor includes a color filter array used to capture the NIR image frame and the visible light image frame at the same time. In other embodiments, in order to generate an NIR image frame, the computing device can cause or control a filter to selectively pass the NIR light"); and
a memory storing instructions that, when executed by the processor, cause the electronic device (See Tyan: Fig. 6, and [0095], "The computing device has processing circuitry, such as the illustrated processor 640, and computer readable medium 642 storing a set of instructions 644, 646, 648, 650. The computer readable medium 642 can, for example, include read-only memory (ROM), random-access memory (RAM), electrically erasable programmable read-only memory (EEPROM), Flash memory, a solid state drive, and/or discrete data register sets. The computing device illustrated by FIG. 6 can form part of the imaging device having the image sensor, such as the processor 640 including part of the control circuitry illustrated by FIG. 1") to:
generate N channel images based on the multispectral signal obtained from a plurality of channels of the multispectral sensor, wherein N is an integer of at least four (See Tyan: Figs. 3A-B, and [0081], "Four images, R, G, B, and NIR, are extracted from the corresponding channels in the raw image. The polarized signal can be extracted from the two images captured with and without polarization illumination, such as two NI R images captured with and without polarization illumination. Compared to other systems addressing the special lighting issue in an operation room, the imaging apparatus, as described herein, can be very simple and uses a compact way of providing high sensitivity in detecting soft tissue signal(s). The imaging system can capture color reflectance and NIR images in real time, with the color reflectance image and NIR image being well-aligned because both are captured at the same time". Note that 4 channel images are mapped to N-channel images with N = 4 >0);
select at least one first channel image corresponding to a visible wavelength band from among the N channel images (See Tyan: Fig. 1, and [0069], "TA filter 107 is arranged along the optical pathway, and selectively passes the reflected light in a visible light range and a NIR light range toward the image sensor 104. The filter 107 can include a notch filter or a band pass filter. As a specific example, the filter 107 includes a first bandpass filter to selectively pass visible light or wavelengths"; and [0098], "Once the plurality of NIR image frames are captured, the computing device adjusts the first polarizer and second polarizer to the first polarization angle and the second polarization angle, and causes the filter to selectively pass the visible light and to generate a visible light image frame. The computing device repeats the adjustment of the polarizers and captures the plurality of visible image frames using each of the different polarization angles of the set". The visible lights are allowed to pass and generate the visible image, and this is mapped to "select at least one first channel image corresponding to a visible wavelength band from among the N channel images");
generate a RGB reference image (See Tyan: Fig. 17, and [0136], “To process the visible images, the RGB channels of a visible image 1771 are converted into the YCbCr color space, at 1773. At 1774, the CLAHE filter is applied to the luminance component Y, while Cb and Cr channels remain unchanged”) based on the at least one first channel image (See Tyan: Fig. 1, and [0074], "In specific embodiments, the control circuitry 108 collects the image data by capturing first image data using collimated incident light as generated by the first and second polarizer 105, 106 and capturing second image data using non-polarized light from the light source 103. The non-polarized light can be used to capture an image that is used as a reference, which is compared to the other images captured using polarized light. Additionally, the reference (e.g., a reference image captured with non-polarized light) can be used as a baseline for fusing with the other images to form an optimal and/or enhanced image. In some specific embodiments, the captured first and second image data includes still image frames and/or video of the sample 109"; and Figs. 18A-B, and [0139], "FIGS. 18A-18B illustrate an example of an image alignment module, in accordance with various embodiments. The image alignment module is used to align images when image sensor (e.g., camera) motion is introduced during image acquisition. Although the image sensor illustrated herein is shown as being immobile, image sensor motion can occur when the bandpass filters are switched to acquire multi-spectral image, among other reasons. This is a design issue that can be avoided, but may be inevitable for practical clinical use where the image sensor is in motion, such as in constant motion. To estimate such a camera motion, it can be assumed that the relationship between two images (e.g., reference image and target image) is a rigid transformation. The image alignment module can be used or be based on ORB approach to match the corresponding points with each other, and which is used to align image frames between one another and combine a plurality of NIR frames into a single NIR frame and a plurality of visible image frames into a single visible image frame". Note that the reference images are visible images, and the invisible NIR images are the target images, thus, the visible image is mapped to the reference image);
select a second channel image from remaining channel images of the N channel images (See Tyan: Fig. 1, and [0069], "A filter 107 is arranged along the optical pathway, and selectively passes the reflected light in a visible light range and a NIR light range toward the image sensor 104. The filter 107 can include a notch filter or a bandpass filter. As a specific example, the filter 107 includes a first bandpass filter to selectively pass visible light or wavelengths and a second bandpass filter to selectively pass NIR light or wavelengths"; and [0072], "In various embodiments, a plurality of NIR images and a plurality of visible light images, using different angled polarized light, can be collected. The control circuitry 108 can sequence through the different polarization angles for NIR images followed by visible light images, and in other embodiments, can collect an NIR image and a visible light image (prior to rotating to different polarized angles) and sequences through the different polarization angles". Note that a plurality of NIR images is mapped to the second channel images), wherein the second channel image corresponds to a reflection wavelength range of a target substance of the measurement-target object (See McQuilkin: Fig. 1, and [0155], “In block 23, the spectral information of the target substance and optionally the background (if desired) is then analyzed via suitable signal processing and spectral analysis techniques to uniquely identify one or more, specific spectral wavelengths that may be used to distinguish the target substance from the background. In preferred modes of practice, at least two specific spectral wavelengths are identified. More preferably, 2 to 500, even more preferably 2 to 100, even more preferably 2 to 50, and even more preferably 2 to 15 specific wavelengths are identified”; [0270], “In another aspect, the present invention relates to a method to analyze substances present in water. As discussed herein, the spectral absorption band of water near 980 nm is narrow. This characteristic permits the identification of water with a filter element sensitive near 980 nm. Adjacent bands, outside of the water absorption band, are available to detect impurities, contaminants or formulations within the water. Examples of such uses include but are not limited to, pollution detection, on-site water analysis, aerial inspection of water quality, beverage processing, wastewater treatment, drinking water treatment, biosecurity at reservoirs, toxicity analyses, presence of organic material, presence of organisms, groundwater analysis and underwater exploration or mining”; and Figs, 25A-E, and [0456], “Referring to FIGS. 25A to 25E, the principles of the present invention are illustrated by an example having two glass vessels partially filled with water that are within the field of view of a spectral imaging system of the present invention. The system is used to detect the presence of water in the two vessels. In each of FIGS. 25A through 25E, the vessel on the left is a round drinking glass. The vessel on the right is a square glass flower vase with a bottom of non-uniform thickness. It is desired to generate a display image that shows the position of the water within each vessel, as well as an indication of the spatial position of objects within the field of view. FIG. 25A is a ‘full spectrum’ image taken with the narrower visible spectrum filter removed (typically included with digital cameras); FIG. 25B is an NIR detection image, sensitive to a narrow bandwidth centered on the absorption peak of water at 980 nm. FIG. 25C is a spectral reference image centered in an area of the spectrum which exhibits little spectral absorption for water at 766 nm. FIG. 25D is a ratio image representing the reference image divided by the NIR image. FIG. 25E is a display image created from the reference image, those regions of the ratio image where the water is present, and a custom colormap to highlight the position of water. A reference spectrum for water is shown in FIG. 2A”. Note that the target object is water here as an example, and specific wavelength signals are selected to detect “water”);
generate object information of the measurement-target object (See Xiao: Fig. 1, and [0017], "It should be understood that in this embodiment of this application, due to different reflectance of near-infrared light for different types of objects, different objects in the third image include different detail information; therefore, an image region in which a photographed object of a target type (for example, green plants or distant mountains) is located may be obtained from the third image based on the third mask (for example, a semantic segmentation mask); and fusion processing is then performed on the image region and the fourth image, to improve local detail information of the fused image"; and [0296], "It should be understood that for part of scenery (for example, green plants or distant mountains), the NIR image has more detail information. For example, because the green scene has a higher reflectance for near-infrared light, the NIR image has more detail information of the green plants than the RGB image. For example, near-infrared light has longer wavelength than visible light, and the near-infrared light features a stronger diffraction capability. In addition, light with longer wavelength has stronger penetration, and therefore a captured image has a stronger sense of transparency. Compared with the RGB image, the NIR image includes more detail information (for example, texture information of distant mountains) for distant scenery (for example, distant mountains). A local region (for example, an image region with more detail information) may be selected from the NIR image and fused with the RGB image, so as to enhance local detail information of the fused image and improve the image quality of the fused image". Note that the third image is NIR image, which is equivalent to the second image of the instant application, and the NIR image gives out more information of the objects for the fused image to enhance the final image, thus this is mapped to "generate object information by analyzing the second channel image") by calculating normalized pixel values of the measurement-target object in the second channel image relative to all pixel values of the second channel image (See McQuilkin: Fig. 1, and [0147], “An illustrative embodiment of a spectral imaging system 10 of the present invention is schematically shown in FIG. 1. Spectral imaging system 10 includes set-up features and operational features. The set-up features, customized for each chosen target substance, involve obtaining the electromagnetic spectrum for the chosen target substance, selecting unique spectral wavelength(s) to identify the substance, selecting corresponding filter elements, and designing a unique target algorithm to reliably identify the target. The operational features include both equipment and processing components. The operational equipment comprises a spectral filter card 106 and a multi-camera array 111 having image capturing elements (such as CCD sensors). The operational processing components may include the following processing steps: image acquisition, image alignment, image normalization for variations in spectral parameters, computation of the presence and/or quantity of the target substance, generation of a colormap, and generation and display of an output image showing the result of the detection procedure. The operational blocks are processing operations that may be executed at or near real time as the multi-camera system is operating and the target substance is within the field of view. While an advantage of the present invention is real-time operation, the present invention also encompasses modes of operation in non-real time mode, batch mode, or other slow or delayed processing operation. Additionally, the simultaneous image acquisition permits freeze-motion images to be easily obtained for moving targets”; [0183], “Once the illumination spectrum is known, the intensity of each acquired image may be normalized in optional block 34 to take the spectral features of the given illumination into account. A variation on the automated identification of the illumination source would include a user interface (i.e., software, switch or button) to manually identify the type of illumination as shown by optional block 27”; [0235], “For example, when the principles of the present invention are used to detect water as described herein and depending upon other materials in the background, a simple ratio between the spectra sampled at two different wavelengths would be quite effective to detect water in a field of view of a spectral imaging system. If spectral information for a particular image pixel demonstrates such a ratio, the system may accurately conclude that water is present in the captured image at that pixel location”; and [0267], “In one aspect, the present invention relates to a method to derive information about water within a target field of view. If the target substance is a volume of water, the selected wavelengths might be one wavelength in which water is absorbed, such as 980 nm, and one wavelength in which water exhibits little absorption, such as 766 nm. Filter elements would contain bandpass filters for each of the selected wavelengths. After image alignment, a simple ratio of image intensities for corresponding pixels would yield an indication of the presence of water”. Note that the image pixel values are normalized, and water as an example, a simple ratio of the pixel values is used to detect the presence of water in the field of views); and
generate the output image by combining the reference image with the object information (See Tyan: Fig. 9, and [0143], "FIG. 19 illustrates an example of an image fusion module, in accordance with various embodiments. Combining near-infrared information with visible images can further enhance the contrast and details and produce more vivid colors. This combination can be achieved using fusion algorithm, based on pyramid transform with a pattern selective approach. FIG. 19 shows the general framework with two source images for simple illustration").
Regarding claim 11, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 10 as outlined above. Further, Tyan teaches that the electronic device of claim 10, wherein the at least one first channel image comprises a channel-R image, a channel-G image, and a channel-B image (See Tyan: Figs. 3A-B, and [0082], "FIG. 3A, more specifically, illustrates an example color filter array 316. As shown by the insert, the color filter array 316 includes an array of four channels 315, 317, 319, 321. The four channels include NIR 315, Red (R) 317, Blue (B) 319, and Green (G) 321. FIG. 3B is a graph 318 showing an example spectral sensitivity for the notch filters which include NIR 315A, R 317A, B 319A, and G 321A. As shown in FIG. 3B, there is crosstalk between the R, G, B, and NIR channels. The color image channels, e.g., R, G and B, are sensitive to the NIR component. The NIR channel may also not be ideal and is sensitive to visible light, an image enhancement technique together with calibration can be used to extract color and NIR imagery from raw images more accurately").
Regarding claim 12, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 10 as outlined above. Further, Xiao teaches that the electronic device of claim 10, wherein the instructions, when executed by the processor, further cause the electronic device to:
set at least one region of interest of the second channel image, based on the input signal received from the input unit (See Xiao: Fig. 1, and [0216], "Because image details of a distant-scene region for the photographed object in the third image are better than image details of a distant-scene region for the photographed object in the fourth image, and image details of a near-scene region for the photographed object in the fourth image are better than image details of a near-scene region for the photographed object in the third image, an image region corresponding to the distant-scene region can be selected from the third image and fused with the fourth image". Note that the region selected in the third (NIR image) is mapped tot at least one region of interest of the second channel image); and
calculate the object information based on pixel values of the at least one region of interest of the second channel image (See Xiao: Fig. 1, and [0253], "For example, the target mask and the third image are multiplied pixel by pixel, and the local image region in the third image may be determined based on pixel values of different regions in the target mask. For example, a local image region in the third image corresponding to an image region with a pixel of 1 in the target mask may be obtained, and the fused image may be obtained through fusion processing on the fourth image and the local image region in the third image". Note that the third image of Xiao is the NIR image, mapped to the second channel image, and the pixel values are used to determine the region/object mask information, which is mapped to the object information).
Regarding claim 13, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 12 as outlined above. Further, Xiao teaches that the electronic device of claim 12, wherein the instructions, when executed by the processor, further cause the electronic device to:
calculate the object information based on an average value of the pixel values of the at least one region of interest (See Xiao: Fig. 1, and [0131], "Exposure is an exposure time, Aperture is an aperture size, lso is light sensitivity, and Luma is an average value of Yin an XYZ space of an image". Note that the average Luma Y in XYZ space of the image is mapped to the average value of the pixel values of the at least one region of interest).
Regarding claim 15, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 10 as outlined above. Further, Xiao teaches that the electronic device of claim 10, wherein the instructions, when executed by the processor, further cause the electronic device to:
extract an edge image extracted from the second channel image (See Xiao: Fig. 12, and [0435], "Optionally, the edge difference image may be obtained based on a union of the high-frequency information of the NIR image and the high-frequency information of the RGB image". Note that the edge difference image is mapped to the edge image); and
add the edge image to the object information (See Xiao: Fig. 12, and [0446], "In this embodiment of this application, the high-frequency information in the NIR image and the RGB image is obtained, so as to obtain the edge difference image. The edge difference image includes all the high-frequency information in the NIR image and the RGB image, and the edge difference image may be used for obtaining the local image region from both the NIR image and the RGB image. Registration processing is then performed on the two local regions, for example, registration processing is performed on the local region in the NIR image by using the local image region in the RGB image as a benchmark, so as to implement registration between the high-frequency information in the NIR image and the high-frequency information in the RGB image, thereby avoiding ghosting in the fused image to some extent". Note that the edge information registration is mapped to add the edge image to the object information).
Regarding claim 16, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 15 as outlined above. Further, Xiao teaches that the electronic device of claim 15, wherein the instructions, when executed by the processor, further cause the electronic device to:
combine the edge image with the reference image to generate the output image, wherein the edge image and the reference image are combined by performing at least one of:
adding the edge image to the reference image (See Xiao: Fig. 8, and [0375], "In this embodiment of this application, the electronic device may include the first camera module and the second camera module. The first camera module is a near-infrared camera module or an infrared camera module. The first camera module may capture the N IR Raw image and the second camera module may capture the RGB Raw image. Image processing is performed on the NIR Raw image and the RGB Raw image to obtain the NIR image and the RGB image, respectively. Fusion processing is performed on the NIR image and the RGB image based on at least two masks to obtain a fused image. Because the NIR image is a near-infrared image or an infrared image, the NIR image may include information that cannot be obtained from the RGB image. Through fusion processing on the NIR image and the RGB image, multi-spectrum information fusion on near-infrared light image information and visible light image information can be implemented to make the fused image include more detail information. In addition, fusion processing on the NIR image and the RGB image is performed based on the at least two masks, and image enhancement can be performed from at least two aspects such as sharpness, ghosting removal, or local details. In this way, image enhancement is implemented forth RGB image obtained by the second camera module (for example, the main-camera module) to enhance the detail information in the image and improve the image quality". Note that the edge different image is registered locally, and it is fused into the RGB image (reference image) to generate the final enhanced image, this is mapped to adding the edge image to the reference image); and
subtracting the edge image from the reference image (See Xiao: Fig. 8, and [0375], "In this embodiment of this application, the electronic device may include the first camera module and the second camera module. The first camera module is a near-infrared camera module or an infrared camera module. The first camera module may capture the N IR Raw image and the second camera module may capture the RGB Raw image. Image processing is performed on the NIR Raw image and the RGB Raw image to obtain the NIR image and the RGB image, respectively. Fusion processing is performed on the NIR image and the RGB image based on at least two masks to obtain a fused image. Because the NIR image is a near-infrared image or an infrared image, the NIR image may include information that cannot be obtained from the RGB image. Through fusion processing on the NIR image and the RGB image, multi-spectrum information fusion on near-infrared light image information and visible light image information can be implemented to make the fused image include more detail information. In addition, fusion processing on the NIR image and the RGB image is performed based on the at least two masks, and image enhancement can be performed from at least two aspects such as sharpness, ghosting removal, or local details. In this way, image enhancement is implemented forthe RGB image obtained by the second camera module (for example, the main-camera module) to enhance the detail information in the image and improve the image quality". Note that the edge different image is registered locally, and it is fused into the RGB image (reference image) to generate the final enhanced image using two masks, when the mask from the NIR image is used, i.e., the edge image from the NIR image is fused into the RGB reference image, the edge image from the RGB reference image is replaced by the NIR edge image, which is mapped to subtracting the edge image from the reference image).
Regarding claim 17, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 10 as outlined above. Further, Tyan teaches that the electronic device of claim 10, wherein the instructions, when executed by the processor, further cause the electronic device to:
select the second channel image based on the input signal received from the input unit (See Tyan: Fig. 1, and [0069], "A filter 107 is arranged along the optical pathway, and selectively passes the reflected light in a visible light range and a NIR light range toward the image sensor 104. The filter 107 can include a notch filter or a bandpass filter. As a specific example, the filter 107 includes a first bandpass filter to selectively pass visible light or wavelengths and a second bandpass filter to selectively pass NIR light or wavelengths"; and [0072], "In various embodiments, a plurality of NIR images and a plurality of visible light images, using different angled polarized light, can be collected. The control circuitry 108 can sequence through the different polarization angles for NIR images followed by visible light images, and in other embodiments, can collect an NIR image and a visible light image (prior to rotating to different polarized angles) and sequences through the different polarization angles". Note that a plurality of NIR images is mapped to the second channel images).
Regarding claim 20, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 10 as outlined above. Further, Xiao teaches that the electronic device of claim 10, wherein the input unit is disposed on the display (See Xiao: Fig. 14, and [0452], "A GUI shown in (a) of FIG. 14 is a desktop 910 of the electronic device. After the electronic device detects a tap operation by a user on an icon 920 of a camera application (application, APP) on the desktop 910, the camera application can be started, and another GUI shown in (b) of FIG.14 is displayed. The GUI shown in (b) of FIG. 14 may be a display screen of the camera APP in photographing mode and the GUI may include a shooting screen 930. The shooting screen 930 may include a viewfinder frame 931 and a control. For example, the shooting screen 930 may include a control 932 for indicating to take a photo and a control 933 for turning on an infrared flash. In a preview state, the viewfinder frame 931 may display a preview image in real time. The preview state may be after the user turns on the camera but before the user presses a photo/record button. In this case, the viewfinder frame may display a preview image in real time". Note that user inputs are displayed on the mobile device display screen, which is mapped to "the input unit is disposed on the display"), and
wherein the input unit is further configured to receive based on a touch input generated by a user on the display displaying the output image (See Xiao: Fig. 14, and [0453], "After the electronic device detects an operation of tapping by the user on the control 933 for indicating to turn on the infrared flash, a shooting screen shown in (c) of FIG. 14 is displayed. When the infrared flash is turned on, images may be captured by a main-camera module and a near-infrared camera module, and fusion processing is performed on the captured images by using the image processing method provided in the embodiments of this application, so as to output a processed fused image". Note that user pushes the flash on button, and the flash is on, which is mapped to "receive based on a touch input generated by a user on the display displaying the output image").
Claims 8 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Tyan, etc. (US 20220125280 A1) in view of Xiao, etc. (US 20240185389 A1), further in view of McQuilkin, etc. (US 20170089761 A1) and Bernstein, etc. (US 20050180651 A1).
Regarding claim 8, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 1 as outlined above. However, Tyan, modified by Xiao and McQuilkin, fails to explicitly disclose that the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: select the second channel image based on an application example of the object.
However, Bernstein teaches that the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: select the second channel image based on an application example of the object (See Bernstein: Figs. 3-4, and [0020], "If a suitable window band is not available, the normalization can still be extracted directly from the standard deviation curve. Two bands (.lambda..sub.2>.lambda.. sub.1) are selected which are outside of any water absorption region, insuring that the atmospheric extinction is due primarily to the aerosols"; and [0047], "It is important to screen for and eliminate anomalous pixel spectra from the end member selection process. This includes pixels containing opaque clouds, thin cirrus clouds, and "bad" pixels containing sensor artifacts. Opaque clouds may be recognized using one of two methods, depending on the available sensor bands. If bands are available in either of the 940 nm or 1140 nm water vapor absorption bands, then opaque clouds can be recognized through anomalously small absorption depressions, as the clouds reside above most of the water vapor column. If the water bands are not available, then clouds can be recognized through a whiteness-brightness test; they are spectrally flat (white) and exhibit a high reflectance (bright). Thin cirrus is most easily flagged through an excess signal (cloud back scattering) in the very dark 1380 nm water absorption band. Cirrus clouds occur at much higher altitudes than other clouds, and thus are detectable even in very strongly absorbing water bands. Bad pixels are recognized through the presence of anomalously high (saturated) or low (negative) spectral channels. The screening thresholds for these types of anomalous pixels can be set conservatively. Since a reasonably large number of end members are selected, it does not matter if a few legitimate spectra are eliminated in the screening process". Note that the multispectral bands (NI R) are selected based on the scene/object properties for the NIR images, and this is mapped to select the second channel image based on an application example of the object).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Tyan to have the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: select the second channel image based on an application example of the object as taught by Bernstein in order to improve the accuracy of determining the atmospheric aerosol optical properties (See Bernstein: Fig. 2, and [0009], "The invention includes methods for retrieving the wavelength-dependent optical depth of the aerosol or haze and molecular absorbers. The aerosol optical depth retrieval method of the current invention, unlike prior art methods, does not require the presence of dark pixels. The retrieved optical depth information can be utilized to improve the accuracy of methods that use first-principles modeling. In particular, it can be used to set the optical depth of a model aerosol when dark pixels are unavailable, or to select from among alternative model aerosols to provide consistency between optical depths retrieved from a dark pixel method and from the current invention"). Tyan teaches a method and system that may generate an enhanced image by fusing the visible light images with the NIR image captured by the multispectral sensors; while Bernstein teaches a system and method that may select the spectrum bands for the NIR images based on the scene or the objects to be imaged in order to improve accuracy of the fused images. Therefore, it is obvious to one of ordinary skill in the art to modify Tyan by Bernstein to select spectrum band for NIR images in order to obtain enhanced fusion images. The motivation to modify Tyan by Bernstein is "Use of known technique to improve similar devices (methods, or products) in the same way".
Regarding claim 19, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 10 as outlined above. Further, Bernstein teaches that the electronic device of claim 10, wherein the object information comprises information about at least one ingredient comprised in the object (See Bernstein: Figs. 3-4, and [0008], "More sophisticated prior art methods are based on first-principles computer modeling. These methods require extensive, and often time-consuming, calculations with a radiative transfer code, such as MODTRAN [Berk et al., 1998], in which A, B and Care computed for a wide range of atmospheric conditions (aerosol and water column amounts and different surface reflectance values). The calculations may be performed for each image to be analyzed, or may be performed ahead of time and stored in large look-up tables". Note that the aerosol and water column amounts in the atmosphere is mapped to the object ingredient).
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Tyan, etc. (US 20220125280 A1) in view of Xiao, etc. (US 20240185389 A1), further in view of McQuilkin, etc. (US 20170089761 A1) and Lin, etc. (US 20130063624 A1).
Regarding claim 9, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 1 as outlined above. However, Tyan, modified by Xiao and McQuilkin, fails to explicitly disclose that the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: generate the N channel images by demosaicing the signals obtained from the plurality of channels of the multispectral sensor.
However, Lin teaches that the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
generate the N channel images by demosaicing the signals obtained from the plurality of channels of the multispectral sensor (See Lin: Fig. 1, and [0129], "The second system studied was a conventional spectral imaging system. The sensor was simulated with a 6-channel multispectral filter array, and demosaiced with a generic 6-channel demosaicing algorithm. The filters used are identical to the ones used in the first system. In contrast to the first system, the second system only requires one capture, but sacrifices some spatial resolution in exchange. For example, a 50-megapixel multispectral camera may use a mosaiced multiband acquisition system with 6-channels". Note that one multispectral sensor capturing one image and demosaicing the image into 6 channels is mapped to generate the N channel images by demosaicing the signals obtained from the plurality of channels of the multispectral sensor).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Tyan to have the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: generate the N channel images by demosaicing the signals obtained from the plurality of channels of the multispectral sensor as taught by Lin in order to increase the capture efficiency (See Lin: Fig. 1, and [0077], "A TFD having an increased number of captured channels may eliminate the necessity of a color filter array and therefore reduces the overall complexity of the system. Thus, by using a tunable spectral imaging sensor, it may be possible to develop a reconfigurable spectral imaging system that adapts to the content of the scene, increasing capture efficiency"). Tyan teaches a method and system that may generate an enhanced image by fusing the visible light images with the NIR image captured by the multispectral sensors; while Lin teaches a system and method that may capture one image with multispectral sensors and de mosaic the captured images into several channels images according to the spectrum filters. Therefore, it is obvious to one of ordinary skill in the art to modify Tyan by Lin to capture one image and demosaic it into several channel images. The motivation to modify Tyan by Lin is "Use of known technique to improve similar devices (methods, or products) in the same way".
Regarding claim 18, Tyan, Xiao, and McQuilkin teach all the features with respect to claim 10 as outlined above. Further, Lin teaches that the electronic device of claim 10, wherein the instructions, when executed by the processor, further cause the electronic device to:
generate the N channel images by demosaicing the multispectral signal obtained from the plurality of channels of the multispectral sensor (See Lin: Fig. 1, and [0129], "The second system studied was a conventional spectral imaging system. The sensor was simulated with a 6-channel multispectral filter array, and demosaiced with a generic 6-channel demosaicing algorithm. The filters used are identical to the ones used in the first system. In contrast to the first system, the second system only requires one capture, but sacrifices some spatial resolution in exchange. For example, a 50-megapixel multispectral camera may use a mosaiced multiband acquisition system with 6-channels". Note that one multispectral sensor capturing one image and demosaicing the image into 6 channels is mapped to generate the N channel images by demosaicing the signals obtained from the plurality of channels of the multispectral sensor).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618
Prosecution Insights