DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Claims 1-20 are pending under this Office action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 10-17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Tyan, etc. (US 20220125280 A1) in view of Xiao, etc. (US 20240185389 A1).
Regarding claim 1, Tyan teaches that an apparatus for acquiring images (See Tyan: Fig. 1, and [0065], “Turning now to the figures, FIG. 1 illustrates an example of an apparatus for imaging, in accordance with various embodiments. The apparatus can be used for imaging of soft tissue using cross-angled polarized light, as further described herein”), the apparatus comprising:
a multispectral sensor configured to sense light reflected from an object (See Tyan: Fig. 1, and [0070], “The image sensor 104, which includes circuitry, collects light reflected from the sample 109 in response to the passed first polarization light and second polarization light in the visible and/or MR light range or wavelengths. As further described herein, a plurality of images can be captured at each of the visible light range and the NIR light range, and while the first and second polarizers 105, 106 are at different angles. The image sensor 104 can include a multi-channel sensor, such as a multi-channel camera”. Note that the multi-channel image sensor is mapped to the multispectral image sensor);
one or more processors (See Tyan: Fig. 1, and [0076], “As may be appreciated, the processing circuitry 110 (sometimes referred to as “a processor”) can be implemented as a multi-core processor or a processor circuit implemented as a set of processor circuits integrated as a chip set. The processing circuitry 110 can thereby include a single, or multiple computer circuits including memory circuitry for storing and accessing the firmware or program code to be accessed or executed as instructions to perform the related operation(s)”); and
a memory storing instructions that, when executed by the one or more processors, cause the apparatus (See Tyan: Fig. 6, and [0095], “The computing device has processing circuitry, such as the illustrated processor 640, and computer readable medium 642 storing a set of instructions 644, 646, 648, 650. The computer readable medium 642 can, for example, include read-only memory (ROM), random-access memory (RAM), electrically erasable programmable read-only memory (EEPROM), Flash memory, a solid state drive, and/or discrete data register sets. The computing device illustrated by FIG. 6 can form part of the imaging device having the image sensor, such as the processor 640 including part of the control circuitry illustrated by FIG. 1”) to:
generate N channel images based on signals obtained from a plurality of channels of the multispectral sensor, N being a positive integer greater than zero (See Tyan: Figs. 3A-B, and [0081], “Four images, R, G, B, and NIR, are extracted from the corresponding channels in the raw image. The polarized signal can be extracted from the two images captured with and without polarization illumination, such as two NIR images captured with and without polarization illumination. Compared to other systems addressing the special lighting issue in an operation room, the imaging apparatus, as described herein, can be very simple and uses a compact way of providing high sensitivity in detecting soft tissue signal(s). The imaging system can capture color reflectance and NIR images in real time, with the color reflectance image and NIR image being well-aligned because both are captured at the same time”. Note that 4 channel images are mapped to N-channel images with N = 4 >0);
select at least one first channel image corresponding to a visible wavelength band from among the N channel images (See Tyan: Fig. 1, and [0069], “TA filter 107 is arranged along the optical pathway, and selectively passes the reflected light in a visible light range and a NIR light range toward the image sensor 104. The filter 107 can include a notch filter or a bandpass filter. As a specific example, the filter 107 includes a first bandpass filter to selectively pass visible light or wavelengths”; and [0098], “Once the plurality of NIR image frames are captured, the computing device adjusts the first polarizer and second polarizer to the first polarization angle and the second polarization angle, and causes the filter to selectively pass the visible light and to generate a visible light image frame. The computing device repeats the adjustment of the polarizers and captures the plurality of visible image frames using each of the different polarization angles of the set”. The visible lights are allowed to pass and generate the visible image, and this is mapped to “select at least one first channel image corresponding to a visible wavelength band from among the N channel images”);
generate a reference image based on the at least one first channel image (See Tyan: Fig. 1, and [0074], “] In specific embodiments, the control circuitry 108 collects the image data by capturing first image data using collimated incident light as generated by the first and second polarizer 105, 106 and capturing second image data using non-polarized light from the light source 103. The non-polarized light can be used to capture an image that is used as a reference, which is compared to the other images captured using polarized light. Additionally, the reference (e.g., a reference image captured with non-polarized light) can be used as a baseline for fusing with the other images to form an optimal and/or enhanced image. In some specific embodiments, the captured first and second image data includes still image frames and/or video of the sample 109”; and Figs. 18A-B, and [0139], “FIGS. 18A-18B illustrate an example of an image alignment module, in accordance with various embodiments. The image alignment module is used to align images when image sensor (e.g., camera) motion is introduced during image acquisition. Although the image sensor illustrated herein is shown as being immobile, image sensor motion can occur when the bandpass filters are switched to acquire multi-spectral image, among other reasons. This is a design issue that can be avoided, but may be inevitable for practical clinical use where the image sensor is in motion, such as in constant motion. To estimate such a camera motion, it can be assumed that the relationship between two images (e.g., reference image and target image) is a rigid transformation. The image alignment module can be used or be based on ORB approach to match the corresponding points with each other, and which is used to align image frames between one another and combine a plurality of NIR frames into a single NIR frame and a plurality of visible image frames into a single visible image frame”. Note that the reference images are visible images, and the invisible NIR images are the target images, thus, the visible image is mapped to the reference image);
select a second channel image from remaining channel images of the N channel images, the remaining channel images corresponding to remaining channels of the plurality of channels of the multispectral sensor distinct from first channels corresponding to the at least one first channel image (See Tyan: Fig. 1, and [0069], “A filter 107 is arranged along the optical pathway, and selectively passes the reflected light in a visible light range and a NIR light range toward the image sensor 104. The filter 107 can include a notch filter or a bandpass filter. As a specific example, the filter 107 includes a first bandpass filter to selectively pass visible light or wavelengths and a second bandpass filter to selectively pass NIR light or wavelengths”; and [0072], “In various embodiments, a plurality of NIR images and a plurality of visible light images, using different angled polarized light, can be collected. The control circuitry 108 can sequence through the different polarization angles for NIR images followed by visible light images, and in other embodiments, can collect an NIR image and a visible light image (prior to rotating to different polarized angles) and sequences through the different polarization angles”. Note that a plurality of NIR images is mapped to the second channel images);
generate object information by analyzing the second channel image;
combine the reference image and the object information to generate an output image (See Tyan: Fig. 9, and [0143], “FIG. 19 illustrates an example of an image fusion module, in accordance with various embodiments. Combining near-infrared information with visible images can further enhance the contrast and details and produce more vivid colors. This combination can be achieved using fusion algorithm, based on pyramid transform with a pattern selective approach. FIG. 19 shows the general framework with two source images for simple illustration”); and
display the output image to a user (See Tyan: Fig. 26, and [0165], “FIG. 26 illustrates example polarized images with fusion of enhancement and detection for features. More specifically, FIG. 26 illustrates fusion results from both visible and NIR for image contrast enhancement and object segmentation are demonstration, where neither the visible nor the NIR image alone provides complete information but the combined image does. This combined result can be rendered to surgeons as the final output from the system. Image 2671 is an enhanced polarized VIS image, image 2673 is an enhanced polarized NIR image, and image 2675 is fused image with nerve enhanced. Image 2672 is an enhanced polarized VIS image with nerve detected, image 2674 is an enhanced polarized NIR image with nerve detected, and image 2676 is fused image with nerve detected”. Note that the display of the fused images 2675 and 2676 is mapped to display the output image to a user).
However, Tyan fails to explicitly disclose that generate object information by analyzing the second channel image.
However, Xiao teaches that generate object information by analyzing the second channel image (See Xiao: Fig. 1, and [0017], “It should be understood that in this embodiment of this application, due to different reflectance of near-infrared light for different types of objects, different objects in the third image include different detail information; therefore, an image region in which a photographed object of a target type (for example, green plants or distant mountains) is located may be obtained from the third image based on the third mask (for example, a semantic segmentation mask); and fusion processing is then performed on the image region and the fourth image, to improve local detail information of the fused image”; and [0296], “It should be understood that for part of scenery (for example, green plants or distant mountains), the NIR image has more detail information. For example, because the green scene has a higher reflectance for near-infrared light, the NIR image has more detail information of the green plants than the RGB image. For example, near-infrared light has longer wavelength than visible light, and the near-infrared light features a stronger diffraction capability. In addition, light with longer wavelength has stronger penetration, and therefore a captured image has a stronger sense of transparency. Compared with the RGB image, the NIR image includes more detail information (for example, texture information of distant mountains) for distant scenery (for example, distant mountains). A local region (for example, an image region with more detail information) may be selected from the NIR image and fused with the RGB image, so as to enhance local detail information of the fused image and improve the image quality of the fused image”. Note that the third image is NIR image, which is equivalent to the second image of the instant application, and the NIR image gives out more information of the objects for the fused image to enhance the final image, thus this is mapped to “generate object information by analyzing the second channel image”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Tyan to have generate object information by analyzing the second channel image as taught by Xiao in order to enhance the detailed information in the image (See Xiao: Fig. 1, and [0175], “Through fusion processing on the third image and the fourth image, multi-spectrum information fusion on near-infrared light image information and visible light image information can be implemented to make the fused image include more detail information. In addition, fusion processing is performed on the third image and the fourth image based on the at least two masks, and image enhancement can be performed from at least two aspects such as sharpness, ghosting removal, or local details. In this way, image enhancement is implemented for the second image obtained by the second camera module (for example, the main-camera module) to enhance the detail information in the image and improve the image quality”). Tyan teaches a method and system that may generate an enhanced image by fusing the visible light images with the NIR image captured by the multispectral sensors; while Xiao teaches a system and method that may analyze the NIR image to get the object information such as location, mask, textures, shape, etc. and fuse them into the RGB (visible) image to obtain enhanced fusion images. Therefore, it is obvious to one of ordinary skill in the art to modify Tyan by Xiao to generate objection information and fuse them into the RGB image to obtain the enhanced fusion image. The motivation to modify Tyan by Xiao is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 2, Tyan and Xiao teach all the features with respect to claim 1 as outlined above. Further, Tyan teaches that the apparatus of claim 1, wherein the at least one first channel image comprises a channel-R image, a channel-G image, and a channel-B image (See Tyan: Figs. 3A-B, and [0082], “FIG. 3A, more specifically, illustrates an example color filter array 316. As shown by the insert, the color filter array 316 includes an array of four channels 315, 317, 319, 321. The four channels include NIR 315, Red (R) 317, Blue (B) 319, and Green (G) 321. FIG. 3B is a graph 318 showing an example spectral sensitivity for the notch filters which include NIR 315A, R 317A, B 319A, and G 321A. As shown in FIG. 3B, there is crosstalk between the R, G, B, and NIR channels. The color image channels, e.g., R, G and B, are sensitive to the NIR component. The NIR channel may also not be ideal and is sensitive to visible light, an image enhancement technique together with calibration can be used to extract color and NIR imagery from raw images more accurately”).
Regarding claim 3, Tyan and Xiao teach all the features with respect to claim 1 as outlined above. Further, Xiao teaches that the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
calculate the object information based on pixel values of at least one region of interest of the second channel image (See Xiao: Fig. 1, and [0253], “For example, the target mask and the third image are multiplied pixel by pixel, and the local image region in the third image may be determined based on pixel values of different regions in the target mask. For example, a local image region in the third image corresponding to an image region with a pixel of 1 in the target mask may be obtained, and the fused image may be obtained through fusion processing on the fourth image and the local image region in the third image”. Note that the third image of Xiao is the NIR image, mapped to the second channel image, and the pixel values are used to determine the region/object mask information, which is mapped to the object information).
Regarding claim 4, Tyan and Xiao teach all the features with respect to claim 3 as outlined above. Further, Xiao teaches that the apparatus of claim 3, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
calculate the object information based on an average value of the pixel values of the at least one region of interest (See Xiao: Fig. 1, and [0131], “Exposure is an exposure time, Aperture is an aperture size, Iso is light sensitivity, and Luma is an average value of Y in an XYZ space of an image”. Note that the average Luma Y in XYZ space of the image is mapped to the average value of the pixel values of the at least one region of interest).
Regarding claim 5, Tyan and Xiao teach all the features with respect to claim 3 as outlined above. Further, Tyan teaches that the apparatus of claim 3, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
normalize the pixel values of the at least one region of interest with respect to pixel values of the second channel image (See Tyan: Figs. 4A-B, and [0088], “Since images acquired from ultrasound can be generally in shades of grey, soft tissues such as nerves and blood vessels are hyperechoic and appear to be much brighter in the image. These hyperechoic signals can be further enhanced with harmonic imaging and frequency compounding. In addition, ultrasound elastography used to assess the tissue strain and elasticity of the anatomical structures can also be combined with hyperechoic analysis to distinguish soft tissues from surrounding structures, which can make the tissue detection more robust. With all unique tissue characteristics, the imaging apparatus can apply pattern-selective image fusion in conjunction with a contrast normalization algorithm for both ultrasound and NIR images to effectively highlight local image features in a scene that can maximize detection of soft tissues on and/or below the surface”. Note that the normalization algorithm is applied to different type of image before the image fusion processing, such as RGB image and the false NIR image, which is mapped to “normalize the pixel values of the at least one region of interest with respect to pixel values of the second channel image”).
Regarding claim 6, Tyan and Xiao teach all the features with respect to claim 1 as outlined above. Further, Xiao teaches that the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
extract an edge image from the second channel image (See Xiao: Fig. 12, and [0435], “Optionally, the edge difference image may be obtained based on a union of the high-frequency information of the NIR image and the high-frequency information of the RGB image”. Note that the edge difference image is mapped to the edge image); and
add the edge image to the object information (See Xiao: Fig. 12, and [0446], “In this embodiment of this application, the high-frequency information in the NIR image and the RGB image is obtained, so as to obtain the edge difference image. The edge difference image includes all the high-frequency information in the NIR image and the RGB image, and the edge difference image may be used for obtaining the local image region from both the NIR image and the RGB image. Registration processing is then performed on the two local regions, for example, registration processing is performed on the local region in the NIR image by using the local image region in the RGB image as a benchmark, so as to implement registration between the high-frequency information in the NIR image and the high-frequency information in the RGB image, thereby avoiding ghosting in the fused image to some extent”. Note that the edge information registration is mapped to add the edge image to the object information).
Regarding claim 7, Tyan and Xiao teach all the features with respect to claim 6 as outlined above. Further, Xiao teaches that the apparatus of claim 6, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
combine the edge image with the reference image to generate the output image, wherein the edge image and the reference image are combined by performing at least one of:
adding the edge image to the reference image (See Xiao: Fig. 8, and [0375], “In this embodiment of this application, the electronic device may include the first camera module and the second camera module. The first camera module is a near-infrared camera module or an infrared camera module. The first camera module may capture the NIR Raw image and the second camera module may capture the RGB Raw image. Image processing is performed on the NIR Raw image and the RGB Raw image to obtain the NIR image and the RGB image, respectively. Fusion processing is performed on the NIR image and the RGB image based on at least two masks to obtain a fused image. Because the NIR image is a near-infrared image or an infrared image, the NIR image may include information that cannot be obtained from the RGB image. Through fusion processing on the NIR image and the RGB image, multi-spectrum information fusion on near-infrared light image information and visible light image information can be implemented to make the fused image include more detail information. In addition, fusion processing on the NIR image and the RGB image is performed based on the at least two masks, and image enhancement can be performed from at least two aspects such as sharpness, ghosting removal, or local details. In this way, image enhancement is implemented for the RGB image obtained by the second camera module (for example, the main-camera module) to enhance the detail information in the image and improve the image quality”. Note that the edge different image is registered locally, and it is fused into the RGB image (reference image) to generate the final enhanced image, this is mapped to adding the edge image to the reference image); and
subtracting the edge image from the reference image (See Xiao: Fig. 8, and [0375], “In this embodiment of this application, the electronic device may include the first camera module and the second camera module. The first camera module is a near-infrared camera module or an infrared camera module. The first camera module may capture the NIR Raw image and the second camera module may capture the RGB Raw image. Image processing is performed on the NIR Raw image and the RGB Raw image to obtain the NIR image and the RGB image, respectively. Fusion processing is performed on the NIR image and the RGB image based on at least two masks to obtain a fused image. Because the NIR image is a near-infrared image or an infrared image, the NIR image may include information that cannot be obtained from the RGB image. Through fusion processing on the NIR image and the RGB image, multi-spectrum information fusion on near-infrared light image information and visible light image information can be implemented to make the fused image include more detail information. In addition, fusion processing on the NIR image and the RGB image is performed based on the at least two masks, and image enhancement can be performed from at least two aspects such as sharpness, ghosting removal, or local details. In this way, image enhancement is implemented for the RGB image obtained by the second camera module (for example, the main-camera module) to enhance the detail information in the image and improve the image quality”. Note that the edge different image is registered locally, and it is fused into the RGB image (reference image) to generate the final enhanced image using two masks, when the mask from the NIR image is used, i.e., the edge image from the NIR image is fused into the RGB reference image, the edge image from the RGB reference image is replaced by the NIR edge image, which is mapped to subtracting the edge image from the reference image).
Regarding claim 10, Tyan and Xiao teach all the features with respect to claim 1 as outlined above. Further, Tyan and Xiao teach that an electronic device for acquiring images, the electronic device (See Tyan: Fig. 1, and [0065], “Turning now to the figures, FIG. 1 illustrates an example of an apparatus for imaging, in accordance with various embodiments. The apparatus can be used for imaging of soft tissue using cross-angled polarized light, as further described herein”) comprising:
a multispectral sensor configured to sense light reflected from an object (See Tyan: Fig. 1, and [0070], “The image sensor 104, which includes circuitry, collects light reflected from the sample 109 in response to the passed first polarization light and second polarization light in the visible and/or MR light range or wavelengths. As further described herein, a plurality of images can be captured at each of the visible light range and the NIR light range, and while the first and second polarizers 105, 106 are at different angles. The image sensor 104 can include a multi-channel sensor, such as a multi-channel camera”. Note that the multi-channel image sensor is mapped to the multispectral image sensor);
an input unit configured to receive a user input (See Xiao: Fig. 17, and [0496], “The processor 1101 may be configured to control the electronic device 1100, execute a software program, and process data of the software program. The electronic device 1100 may further include a communications unit 1105, to implement signal input (reception) and output (transmission)”. Note that the communications unit 1105, to implement signal input (reception) is mapped to the input unit);
a display (See Xiao: Fig. 1, and [0151], “The display 194 may be configured to display images or videos”);
a processor configured to generate and display an output image on the display based on a multispectral signal received from the multispectral sensor and an input signal received from the input unit (See Tyan: Fig. 6, and [0096], “The computing device can be used to capture image data of a sample, the image data including and/or being indicative of a plurality of polarized NIR image frames and a plurality of polarized visible light image frames of the sample collected using a plurality of different polarization angles of illumination light and imaging light. For example, at 644, the computing device adjusts the first polarizer and second polarizer to a first polarization angle and a second polarization angle, where the first and second polarization angles are crossed with each other (e.g., orthogonal or slant). The adjustment can include causing or controlling a motor to physically rotate or otherwise change the angle of each polarizer or causing different electrical fields to be applied to the respective polarizers. At 646, the computing device generates an NIR image frame of the sample, and at 648, generates a visible light image frame of the sample while the polarizers are at the first and second polarization angles. In some embodiments, the image sensor includes a color filter array used to capture the NIR image frame and the visible light image frame at the same time. In other embodiments, in order to generate an NIR image frame, the computing device can cause or control a filter to selectively pass the NIR light”); and
a memory storing instructions that, when executed by the processor, cause the electronic device (See Tyan: Fig. 6, and [0095], “The computing device has processing circuitry, such as the illustrated processor 640, and computer readable medium 642 storing a set of instructions 644, 646, 648, 650. The computer readable medium 642 can, for example, include read-only memory (ROM), random-access memory (RAM), electrically erasable programmable read-only memory (EEPROM), Flash memory, a solid state drive, and/or discrete data register sets. The computing device illustrated by FIG. 6 can form part of the imaging device having the image sensor, such as the processor 640 including part of the control circuitry illustrated by FIG. 1”) to:
generate N channel images based on the multispectral signal obtained from a plurality of channels of the multispectral sensor, N being a positive integer greater than zero (See Tyan: Figs. 3A-B, and [0081], “Four images, R, G, B, and NIR, are extracted from the corresponding channels in the raw image. The polarized signal can be extracted from the two images captured with and without polarization illumination, such as two NIR images captured with and without polarization illumination. Compared to other systems addressing the special lighting issue in an operation room, the imaging apparatus, as described herein, can be very simple and uses a compact way of providing high sensitivity in detecting soft tissue signal(s). The imaging system can capture color reflectance and NIR images in real time, with the color reflectance image and NIR image being well-aligned because both are captured at the same time”. Note that 4 channel images are mapped to N-channel images with N = 4 >0);
select at least one first channel image corresponding to a visible wavelength band from among the N channel images (See Tyan: Fig. 1, and [0069], “TA filter 107 is arranged along the optical pathway, and selectively passes the reflected light in a visible light range and a NIR light range toward the image sensor 104. The filter 107 can include a notch filter or a bandpass filter. As a specific example, the filter 107 includes a first bandpass filter to selectively pass visible light or wavelengths”; and [0098], “Once the plurality of NIR image frames are captured, the computing device adjusts the first polarizer and second polarizer to the first polarization angle and the second polarization angle, and causes the filter to selectively pass the visible light and to generate a visible light image frame. The computing device repeats the adjustment of the polarizers and captures the plurality of visible image frames using each of the different polarization angles of the set”. The visible lights are allowed to pass and generate the visible image, and this is mapped to “select at least one first channel image corresponding to a visible wavelength band from among the N channel images”);
generate a reference image based on the at least one first channel image (See Tyan: Fig. 1, and [0074], “] In specific embodiments, the control circuitry 108 collects the image data by capturing first image data using collimated incident light as generated by the first and second polarizer 105, 106 and capturing second image data using non-polarized light from the light source 103. The non-polarized light can be used to capture an image that is used as a reference, which is compared to the other images captured using polarized light. Additionally, the reference (e.g., a reference image captured with non-polarized light) can be used as a baseline for fusing with the other images to form an optimal and/or enhanced image. In some specific embodiments, the captured first and second image data includes still image frames and/or video of the sample 109”; and Figs. 18A-B, and [0139], “FIGS. 18A-18B illustrate an example of an image alignment module, in accordance with various embodiments. The image alignment module is used to align images when image sensor (e.g., camera) motion is introduced during image acquisition. Although the image sensor illustrated herein is shown as being immobile, image sensor motion can occur when the bandpass filters are switched to acquire multi-spectral image, among other reasons. This is a design issue that can be avoided, but may be inevitable for practical clinical use where the image sensor is in motion, such as in constant motion. To estimate such a camera motion, it can be assumed that the relationship between two images (e.g., reference image and target image) is a rigid transformation. The image alignment module can be used or be based on ORB approach to match the corresponding points with each other, and which is used to align image frames between one another and combine a plurality of NIR frames into a single NIR frame and a plurality of visible image frames into a single visible image frame”. Note that the reference images are visible images, and the invisible NIR images are the target images, thus, the visible image is mapped to the reference image);
select a second channel image from remaining channel images of the N channel images, the remaining channel images corresponding to remaining channels of the plurality of channels of the multispectral sensor distinct from first channels corresponding to the at least one first channel image (See Tyan: Fig. 1, and [0069], “A filter 107 is arranged along the optical pathway, and selectively passes the reflected light in a visible light range and a NIR light range toward the image sensor 104. The filter 107 can include a notch filter or a bandpass filter. As a specific example, the filter 107 includes a first bandpass filter to selectively pass visible light or wavelengths and a second bandpass filter to selectively pass NIR light or wavelengths”; and [0072], “In various embodiments, a plurality of NIR images and a plurality of visible light images, using different angled polarized light, can be collected. The control circuitry 108 can sequence through the different polarization angles for NIR images followed by visible light images, and in other embodiments, can collect an NIR image and a visible light image (prior to rotating to different polarized angles) and sequences through the different polarization angles”. Note that a plurality of NIR images is mapped to the second channel images);
generate object information by analyzing the second channel image (See Xiao: Fig. 1, and [0017], “It should be understood that in this embodiment of this application, due to different reflectance of near-infrared light for different types of objects, different objects in the third image include different detail information; therefore, an image region in which a photographed object of a target type (for example, green plants or distant mountains) is located may be obtained from the third image based on the third mask (for example, a semantic segmentation mask); and fusion processing is then performed on the image region and the fourth image, to improve local detail information of the fused image”; and [0296], “It should be understood that for part of scenery (for example, green plants or distant mountains), the NIR image has more detail information. For example, because the green scene has a higher reflectance for near-infrared light, the NIR image has more detail information of the green plants than the RGB image. For example, near-infrared light has longer wavelength than visible light, and the near-infrared light features a stronger diffraction capability. In addition, light with longer wavelength has stronger penetration, and therefore a captured image has a stronger sense of transparency. Compared with the RGB image, the NIR image includes more detail information (for example, texture information of distant mountains) for distant scenery (for example, distant mountains). A local region (for example, an image region with more detail information) may be selected from the NIR image and fused with the RGB image, so as to enhance local detail information of the fused image and improve the image quality of the fused image”. Note that the third image is NIR image, which is equivalent to the second image of the instant application, and the NIR image gives out more information of the objects for the fused image to enhance the final image, thus this is mapped to “generate object information by analyzing the second channel image”); and
generate the output image by combining the reference image with the object information (See Tyan: Fig. 9, and [0143], “FIG. 19 illustrates an example of an image fusion module, in accordance with various embodiments. Combining near-infrared information with visible images can further enhance the contrast and details and produce more vivid colors. This combination can be achieved using fusion algorithm, based on pyramid transform with a pattern selective approach. FIG. 19 shows the general framework with two source images for simple illustration”).
Regarding claim 11, Tyan and Xiao teach all the features with respect to claim 10 as outlined above. Further, Tyan teaches that the electronic device of claim 10, wherein the at least one first channel image comprises a channel-R image, a channel-G image, and a channel-B image (See Tyan: Figs. 3A-B, and [0082], “FIG. 3A, more specifically, illustrates an example color filter array 316. As shown by the insert, the color filter array 316 includes an array of four channels 315, 317, 319, 321. The four channels include NIR 315, Red (R) 317, Blue (B) 319, and Green (G) 321. FIG. 3B is a graph 318 showing an example spectral sensitivity for the notch filters which include NIR 315A, R 317A, B 319A, and G 321A. As shown in FIG. 3B, there is crosstalk between the R, G, B, and NIR channels. The color image channels, e.g., R, G and B, are sensitive to the NIR component. The NIR channel may also not be ideal and is sensitive to visible light, an image enhancement technique together with calibration can be used to extract color and NIR imagery from raw images more accurately”).
Regarding claim 12, Tyan and Xiao teach all the features with respect to claim 10 as outlined above. Further, Xiao teaches that the electronic device of claim 10, wherein the instructions, when executed by the processor, further cause the electronic device to:
set at least one region of interest of the second channel image, based on the input signal received from the input unit (See Xiao: Fig. 1, and [0216], “Because image details of a distant-scene region for the photographed object in the third image are better than image details of a distant-scene region for the photographed object in the fourth image, and image details of a near-scene region for the photographed object in the fourth image are better than image details of a near-scene region for the photographed object in the third image, an image region corresponding to the distant-scene region can be selected from the third image and fused with the fourth image”. Note that the region selected in the third (NIR image) is mapped to t at least one region of interest of the second channel image); and
calculate the object information based on pixel values of the at least one region of interest of the second channel image (See Xiao: Fig. 1, and [0253], “For example, the target mask and the third image are multiplied pixel by pixel, and the local image region in the third image may be determined based on pixel values of different regions in the target mask. For example, a local image region in the third image corresponding to an image region with a pixel of 1 in the target mask may be obtained, and the fused image may be obtained through fusion processing on the fourth image and the local image region in the third image”. Note that the third image of Xiao is the NIR image, mapped to the second channel image, and the pixel values are used to determine the region/object mask information, which is mapped to the object information).
Regarding claim 13, Tyan and Xiao teach all the features with respect to claim 12 as outlined above. Further, Xiao teaches that the electronic device of claim 12, wherein the instructions, when executed by the processor, further cause the electronic device to:
calculate the object information based on an average value of the pixel values of the at least one region of interest (See Xiao: Fig. 1, and [0131], “Exposure is an exposure time, Aperture is an aperture size, Iso is light sensitivity, and Luma is an average value of Y in an XYZ space of an image”. Note that the average Luma Y in XYZ space of the image is mapped to the average value of the pixel values of the at least one region of interest).
Regarding claim 14, Tyan and Xiao teach all the features with respect to claim 12 as outlined above. Further, Tyan teaches that the electronic device of claim 12, wherein the instructions, when executed by the processor, further cause the electronic device to:
normalize the pixel values of the at least one region of interest with respect to pixel values of the second channel image (See Tyan: Figs. 4A-B, and [0088], “Since images acquired from ultrasound can be generally in shades of grey, soft tissues such as nerves and blood vessels are hyperechoic and appear to be much brighter in the image. These hyperechoic signals can be further enhanced with harmonic imaging and frequency compounding. In addition, ultrasound elastography used to assess the tissue strain and elasticity of the anatomical structures can also be combined with hyperechoic analysis to distinguish soft tissues from surrounding structures, which can make the tissue detection more robust. With all unique tissue characteristics, the imaging apparatus can apply pattern-selective image fusion in conjunction with a contrast normalization algorithm for both ultrasound and NIR images to effectively highlight local image features in a scene that can maximize detection of soft tissues on and/or below the surface”. Note that the normalization algorithm is applied to different type of image before the image fusion processing, such as RGB image and the false NIR image, which is mapped to “normalize the pixel values of the at least one region of interest with respect to pixel values of the second channel image”).
Regarding claim 15, Tyan and Xiao teach all the features with respect to claim 10 as outlined above. Further, Xiao teaches that the electronic device of claim 10, wherein the instructions, when executed by the processor, further cause the electronic device to:
extract an edge image extracted from the second channel image (See Xiao: Fig. 12, and [0435], “Optionally, the edge difference image may be obtained based on a union of the high-frequency information of the NIR image and the high-frequency information of the RGB image”. Note that the edge difference image is mapped to the edge image); and
add the edge image to the object information (See Xiao: Fig. 12, and [0446], “In this embodiment of this application, the high-frequency information in the NIR image and the RGB image is obtained, so as to obtain the edge difference image. The edge difference image includes all the high-frequency information in the NIR image and the RGB image, and the edge difference image may be used for obtaining the local image region from both the NIR image and the RGB image. Registration processing is then performed on the two local regions, for example, registration processing is performed on the local region in the NIR image by using the local image region in the RGB image as a benchmark, so as to implement registration between the high-frequency information in the NIR image and the high-frequency information in the RGB image, thereby avoiding ghosting in the fused image to some extent”. Note that the edge information registration is mapped to add the edge image to the object information).
Regarding claim 16, Tyan and Xiao teach all the features with respect to claim 15 as outlined above. Further, Xiao teaches that the electronic device of claim 15, wherein the instructions, when executed by the processor, further cause the electronic device to:
combine the edge image with the reference image to generate the output image, wherein the edge image and the reference image are combined by performing at least one of:
adding the edge image to the reference image (See Xiao: Fig. 8, and [0375], “In this embodiment of this application, the electronic device may include the first camera module and the second camera module. The first camera module is a near-infrared camera module or an infrared camera module. The first camera module may capture the NIR Raw image and the second camera module may capture the RGB Raw image. Image processing is performed on the NIR Raw image and the RGB Raw image to obtain the NIR image and the RGB image, respectively. Fusion processing is performed on the NIR image and the RGB image based on at least two masks to obtain a fused image. Because the NIR image is a near-infrared image or an infrared image, the NIR image may include information that cannot be obtained from the RGB image. Through fusion processing on the NIR image and the RGB image, multi-spectrum information fusion on near-infrared light image information and visible light image information can be implemented to make the fused image include more detail information. In addition, fusion processing on the NIR image and the RGB image is performed based on the at least two masks, and image enhancement can be performed from at least two aspects such as sharpness, ghosting removal, or local details. In this way, image enhancement is implemented for the RGB image obtained by the second camera module (for example, the main-camera module) to enhance the detail information in the image and improve the image quality”. Note that the edge different image is registered locally, and it is fused into the RGB image (reference image) to generate the final enhanced image, this is mapped to adding the edge image to the reference image); and
subtracting the edge image from the reference image (See Xiao: Fig. 8, and [0375], “In this embodiment of this application, the electronic device may include the first camera module and the second camera module. The first camera module is a near-infrared camera module or an infrared camera module. The first camera module may capture the NIR Raw image and the second camera module may capture the RGB Raw image. Image processing is performed on the NIR Raw image and the RGB Raw image to obtain the NIR image and the RGB image, respectively. Fusion processing is performed on the NIR image and the RGB image based on at least two masks to obtain a fused image. Because the NIR image is a near-infrared image or an infrared image, the NIR image may include information that cannot be obtained from the RGB image. Through fusion processing on the NIR image and the RGB image, multi-spectrum information fusion on near-infrared light image information and visible light image information can be implemented to make the fused image include more detail information. In addition, fusion processing on the NIR image and the RGB image is performed based on the at least two masks, and image enhancement can be performed from at least two aspects such as sharpness, ghosting removal, or local details. In this way, image enhancement is implemented for the RGB image obtained by the second camera module (for example, the main-camera module) to enhance the detail information in the image and improve the image quality”. Note that the edge different image is registered locally, and it is fused into the RGB image (reference image) to generate the final enhanced image using two masks, when the mask from the NIR image is used, i.e., the edge image from the NIR image is fused into the RGB reference image, the edge image from the RGB reference image is replaced by the NIR edge image, which is mapped to subtracting the edge image from the reference image).
Regarding claim 17, Tyan and Xiao teach all the features with respect to claim 10 as outlined above. Further, Tyan teaches that the electronic device of claim 10, wherein the instructions, when executed by the processor, further cause the electronic device to:
select the second channel image based on the input signal received from the input unit (See Tyan: Fig. 1, and [0069], “A filter 107 is arranged along the optical pathway, and selectively passes the reflected light in a visible light range and a NIR light range toward the image sensor 104. The filter 107 can include a notch filter or a bandpass filter. As a specific example, the filter 107 includes a first bandpass filter to selectively pass visible light or wavelengths and a second bandpass filter to selectively pass NIR light or wavelengths”; and [0072], “In various embodiments, a plurality of NIR images and a plurality of visible light images, using different angled polarized light, can be collected. The control circuitry 108 can sequence through the different polarization angles for NIR images followed by visible light images, and in other embodiments, can collect an NIR image and a visible light image (prior to rotating to different polarized angles) and sequences through the different polarization angles”. Note that a plurality of NIR images is mapped to the second channel images).
Regarding claim 20, Tyan and Xiao teach all the features with respect to claim 10 as outlined above. Further, Xiao teaches that the electronic device of claim 10, wherein the input unit is disposed on the display (See Xiao: Fig. 14, and [0452], “A GUI shown in (a) of FIG. 14 is a desktop 910 of the electronic device. After the electronic device detects a tap operation by a user on an icon 920 of a camera application (application, APP) on the desktop 910, the camera application can be started, and another GUI shown in (b) of FIG. 14 is displayed. The GUI shown in (b) of FIG. 14 may be a display screen of the camera APP in photographing mode and the GUI may include a shooting screen 930. The shooting screen 930 may include a viewfinder frame 931 and a control. For example, the shooting screen 930 may include a control 932 for indicating to take a photo and a control 933 for turning on an infrared flash. In a preview state, the viewfinder frame 931 may display a preview image in real time. The preview state may be after the user turns on the camera but before the user presses a photo/record button. In this case, the viewfinder frame may display a preview image in real time”. Note that user inputs are displayed on the mobile device display screen, which is mapped to “the input unit is disposed on the display”), and
wherein the input unit is further configured to receive based on a touch input generated by a user on the display displaying the output image (See Xiao: Fig. 14, and [0453], “After the electronic device detects an operation of tapping by the user on the control 933 for indicating to turn on the infrared flash, a shooting screen shown in (c) of FIG. 14 is displayed. When the infrared flash is turned on, images may be captured by a main-camera module and a near-infrared camera module, and fusion processing is performed on the captured images by using the image processing method provided in the embodiments of this application, so as to output a processed fused image”. Note that user pushes the flash on button, and the flash is on, which is mapped to “receive based on a touch input generated by a user on the display displaying the output image”).
Claims 8 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Tyan, etc. (US 20220125280 A1) in view of Xiao, etc. (US 20240185389 A1), further in view of Bernstein, etc. (US 20050180651 A1).
Regarding claim 8, Tyan and Xiao teach all the features with respect to claim 1 as outlined above. However, Tyan, modified by Xiao, fails to explicitly disclose that the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: select the second channel image based on an application example of the object.
However, Bernstein teaches that the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: select the second channel image based on an application example of the object (See Bernstein: Figs. 3-4, and [0020], “If a suitable window band is not available, the normalization can still be extracted directly from the standard deviation curve. Two bands (.lambda..sub.2>.lambda..sub.1) are selected which are outside of any water absorption region, insuring that the atmospheric extinction is due primarily to the aerosols”; and [0047], “It is important to screen for and eliminate anomalous pixel spectra from the end member selection process. This includes pixels containing opaque clouds, thin cirrus clouds, and "bad" pixels containing sensor artifacts. Opaque clouds may be recognized using one of two methods, depending on the available sensor bands. If bands are available in either of the 940 nm or 1140 nm water vapor absorption bands, then opaque clouds can be recognized through anomalously small absorption depressions, as the clouds reside above most of the water vapor column. If the water bands are not available, then clouds can be recognized through a whiteness-brightness test; they are spectrally flat (white) and exhibit a high reflectance (bright). Thin cirrus is most easily flagged through an excess signal (cloud back scattering) in the very dark 1380 nm water absorption band. Cirrus clouds occur at much higher altitudes than other clouds, and thus are detectable even in very strongly absorbing water bands. Bad pixels are recognized through the presence of anomalously high (saturated) or low (negative) spectral channels. The screening thresholds for these types of anomalous pixels can be set conservatively. Since a reasonably large number of end members are selected, it does not matter if a few legitimate spectra are eliminated in the screening process”. Note that the multispectral bands (NIR) are selected based on the scene/object properties for the NIR images, and this is mapped to select the second channel image based on an application example of the object).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Tyan to have the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: select the second channel image based on an application example of the object as taught by Bernstein in order to improve the accuracy of determining the atmospheric aerosol optical properties (See Bernstein: Fig. 2, and [0009], “The invention includes methods for retrieving the wavelength-dependent optical depth of the aerosol or haze and molecular absorbers. The aerosol optical depth retrieval method of the current invention, unlike prior art methods, does not require the presence of dark pixels. The retrieved optical depth information can be utilized to improve the accuracy of methods that use first-principles modeling. In particular, it can be used to set the optical depth of a model aerosol when dark pixels are unavailable, or to select from among alternative model aerosols to provide consistency between optical depths retrieved from a dark pixel method and from the current invention”). Tyan teaches a method and system that may generate an enhanced image by fusing the visible light images with the NIR image captured by the multispectral sensors; while Bernstein teaches a system and method that may select the spectrum bands for the NIR images based on the scene or the objects to be imaged in order to improve accuracy of the fused images. Therefore, it is obvious to one of ordinary skill in the art to modify Tyan by Bernstein to select spectrum band for NIR images in order to obtain enhanced fusion images. The motivation to modify Tyan by Bernstein is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 19, Tyan and Xiao teach all the features with respect to claim 10 as outlined above. Further, Bernstein teaches that the electronic device of claim 10, wherein the object information comprises information about at least one ingredient comprised in the object (See Bernstein: Figs. 3-4, and [0008], “More sophisticated prior art methods are based on first-principles computer modeling. These methods require extensive, and often time-consuming, calculations with a radiative transfer code, such as MODTRAN [Berk et al., 1998], in which A, B and C are computed for a wide range of atmospheric conditions (aerosol and water column amounts and different surface reflectance values). The calculations may be performed for each image to be analyzed, or may be performed ahead of time and stored in large look-up tables”. Note that the aerosol and water column amounts in the atmosphere is mapped to the object ingredient).
Claims 9 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Tyan, etc. (US 20220125280 A1) in view of Xiao, etc. (US 20240185389 A1), further in view of Lin, etc. (US 20130063624 A1).
Regarding claim 9, Tyan and Xiao teach all the features with respect to claim 1 as outlined above. However, Tyan, modified by Xiao, fails to explicitly disclose that the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: generate the N channel images by demosaicing the signals obtained from the plurality of channels of the multispectral sensor.
However, Lin teaches that the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to:
generate the N channel images by demosaicing the signals obtained from the plurality of channels of the multispectral sensor (See Lin: Fig. 1, and [0129], “The second system studied was a conventional spectral imaging system. The sensor was simulated with a 6-channel multispectral filter array, and demosaiced with a generic 6-channel demosaicing algorithm. The filters used are identical to the ones used in the first system. In contrast to the first system, the second system only requires one capture, but sacrifices some spatial resolution in exchange. For example, a 50-megapixel multispectral camera may use a mosaiced multiband acquisition system with 6-channels”. Note that one multispectral sensor capturing one image and demosaicing the image into 6 channels is mapped to generate the N channel images by demosaicing the signals obtained from the plurality of channels of the multispectral sensor).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention was effectively filed to modify Tyan to have the apparatus of claim 1, wherein the instructions, when executed by the one or more processors, further cause the apparatus to: generate the N channel images by demosaicing the signals obtained from the plurality of channels of the multispectral sensor as taught by Lin in order to increase the capture efficiency (See Lin: Fig. 1, and [0077], “A TFD having an increased number of captured channels may eliminate the necessity of a color filter array and therefore reduces the overall complexity of the system. Thus, by using a tunable spectral imaging sensor, it may be possible to develop a reconfigurable spectral imaging system that adapts to the content of the scene, increasing capture efficiency”). Tyan teaches a method and system that may generate an enhanced image by fusing the visible light images with the NIR image captured by the multispectral sensors; while Lin teaches a system and method that may capture one image with multispectral sensors and demosaic the captured images into several channels images according to the spectrum filters. Therefore, it is obvious to one of ordinary skill in the art to modify Tyan by Lin to capture one image and demosaic it into several channel images. The motivation to modify Tyan by Lin is “Use of known technique to improve similar devices (methods, or products) in the same way”.
Regarding claim 18, Tyan and Xiao teach all the features with respect to claim 10 as outlined above. Further, Lin teaches that the electronic device of claim 10, wherein the instructions, when executed by the processor, further cause the electronic device to:
generate the N channel images by demosaicing the multispectral signal obtained from the plurality of channels of the multispectral sensor (See Lin: Fig. 1, and [0129], “The second system studied was a conventional spectral imaging system. The sensor was simulated with a 6-channel multispectral filter array, and demosaiced with a generic 6-channel demosaicing algorithm. The filters used are identical to the ones used in the first system. In contrast to the first system, the second system only requires one capture, but sacrifices some spatial resolution in exchange. For example, a 50-megapixel multispectral camera may use a mosaiced multiband acquisition system with 6-channels”. Note that one multispectral sensor capturing one image and demosaicing the image into 6 channels is mapped to generate the N channel images by demosaicing the signals obtained from the plurality of channels of the multispectral sensor).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GORDON G LIU whose telephone number is (571)270-0382. The examiner can normally be reached Monday - Friday 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona E Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GORDON G LIU/Primary Examiner, Art Unit 2618