Detailed Action
1. Claims 1-20 are pending in this Application.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to amendment
3. Applicant’s response to the last Office Action filed on 11/20/2025 has been entered and made of record.
4. Claims1,3,15,17 and 18 have been amended, and new claim 20 has been added.
.
Response to Argument
5. The Applicant’s argument filed 02/18/2026 is fully consider. For Examiner response see discussion below.
6 The Applicant’s has amended claims 1 and 18 by adding the following limitation
“wherein usage of image processing using the nonvisible light image data differs depending on (a) conditions under which the nonvisible light image data is generated by the imaging unit and (b) conditions under which the processing unit determines to perform image processing using the nonvisible light image data.”, and substantially argue that the applied prior art (OTSU ) does not teach the added limitation.
As to the above argument Examiner partially agree with the Applicant’s argument.
a. Regarding the limitation “wherein usage of image processing using the nonvisible light image data differs depending on (a) conditions under which the nonvisible light image data is generated by the imaging unit”, Examiner respectfully disagree with the Applicant’s argument because OTSU teaches the above limitation. Specifically OTSU teaches the image processing unit 14 is not limited to the resampling process as described above, but various image processes such as luminance gradation correction, detail correction, various noise cancellation filters, color correction, and resolution conversion “ See, page 6 the last two paragraphs.
The above description corresponds to the limitation mentioned above because the specification of this application disclosed : “ In this embodiment, the visible light image sensor 100 and the nonvisible light image sensor 101 are simultaneously driven in the noise reduction and contrast improvement to acquire the pre-correction visible light image data and pre-correction nonvisible light image data at the same time”. See par. [0025] of PGPUB. Thus, OTSUs teach the above limitation.
b. Regarding the limitation “wherein usage of image processing using the nonvisible light image data differs depending on (b) conditions under which the processing unit determines to perform image processing using the nonvisible light image data” the Applicant’s argument is persuasive thus the rejection based on the applied prior art is expressly withdrawn . However, after further search and consideration a new prior art that teaches the added limitations found. Specifically US 8897588 B2 to Wang; et al., disclosed a method of estimation blur degradation of an image (for example it is known that blur can be caused by camera shake ), and correcting the blur degradation of the image by deriving a blur kernel; and deblurring the image by deconvolution using the derived blur kernel (see, Abstract col.4 lines 30-45) . A camera (e.g., which may employ visible or non-visible wavelengths such as infrared frequencies to recognize movement as gestures that do not involve touch. see col14 lines 40-50).
The above description corresponds to the limitation mentioned above because the specification of this application disclosed: “In a case where the determination method is “dark and camera shakes” and the usage is “image stabilization,” image blur may occur in the visible light image data due to the camera shake and the dark object. Therefore, the main control unit 117 instructs the combining unit 109 to perform image stabilizing processing. For the dark object, a blur kernel may be affected by noise in the visible light image data. Therefore, by estimating the blur kernel using the nonvisible light image data, the image stabilization accuracy can be improved.” See par. [0083] of PGPUB .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
7. Claims 1-5,8,13-15 and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over OTSUHI HIRONAO (hereafter OTSU), JP-2010154266A, pub. 07/08/2010, in view of Wang et al., (hereafter Wang), US 8897588 B2, pub. 11/25/2014.
As to claim 1, OTSU teaches An imaging control apparatus (Abstract: an infrared irradiation type image pickup device ) comprising: at least one processor; and a memory coupled to the at least one processor ([0074], the imaging device includes a CPU, a main storage device such as a RAM, and a computer-readable storage medium in the system that stores a program) wherein the memory has instructions that, when executed by the at least one processor, perform operations as a plurality of unites ([0074], the CPU then reads out the control program stored in the storage medium and executes information processing and calculation processing) comprising :
a control unit configured to acquire visible light image data and nonvisible light image data through imaging by an imaging unit ([009], [0018]-[0019], an imaging condition control unit that causes the imaging unit to generate a visible light imaging signal under visible light imaging conditions in which the infrared cut filter is placed at the blocking position, and that causes the imaging unit to generate an infrared mixed imaging signal under
infrared mixed imaging conditions);
a processing unit configured to perform image processing using at least one of the visible light image data and the nonvisible light image data ([00], [0027], an image signal consisting of visible light components, hereinafter, as a visible light image signal RGBxy. The RGBxy is generated by the imaging unit 4 and output to the white balance processing unit 10 and. The white balance processing unit 10 performs white balance processing on the visible light imaging signal RGBxy to generate a white balanced imaging signal wRGBxy. The color-matched image signal processor process the adding the infrared calculation image signal [Axy] to the lightness L component the visible image) ; and
a determining unit configured to determine an imaging scene, and wherein the control unit controls whether or not to perform imaging for the nonvisible light image data by the imaging unit and image processing using the nonvisible light image data in the processing unit according to a determination result of the imaging scene (Claim 2, [0049], the infrared illumination type imaging device according to claim 2, further comprising: a warning unit that issues a warning to an operator when the color change state is detected or when it is determined that the scene has changed; and an operation unit that waits for control switching from the infrared mixed imaging condition to the visible light imaging condition until an operation is performed by the operator);and
wherein usage of image processing using the nonvisible light image data differs depending on (a) conditions under which the nonvisible light image data is generated by the imaging unit ( page 6 last two pars., the image processing unit 14 is not limited to the resampling process as described above, but various image processes such as luminance gradation correction, detail correction, various noise cancellation filters, color correction, and resolution conversion . The corresponded is derived based on the specification of the application.(see par. [0025] of PGPUB) ).
However, it is noted that OTSU does not specifically teach “wherein usage of image processing using the nonvisible light image data differs depending on conditions under which the processing unit determines to perform image processing using the nonvisible light image data”
On the other hand Wang teaches wherein usage of image processing using the nonvisible light image data differs depending on conditions under which the processing unit determines to perform image processing using the nonvisible light image data (Abstract col.4 lines 30-45, a method of estimation blur degradation of an image (for example it is known that blur can be caused by camera shake ), and correcting the blur degradation of the image by deriving a blur kernel; and deblurring the image by deconvolution using the derived blur kernel and teach a camera (e.g., which may employ visible or non-visible wavelengths such as infrared frequencies to recognize movement as gestures that do not involve touch, see col.14 lines 40-50). The corresponded is derived based on the specification of this application(see par. [0083] of PGPUB).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate method of correcting blur by deriving a blur kernel (Point Spread Function) taught by Wang into OTSU.
The suggestion/motivation for doing so would have been to allow user of OTSU to restore the latent sharp image by reversing the exact mathematical degradation by identifying the specific motion or defocus path (the kernel), deconvolution can reverse complex, non-uniform, or large-scale blur to recover fine details.
As to claim 2, OTSU teaches the imaging unit includes a first imaging unit for visible light and a second imaging unit for nonvisible light (claim1,[0009], imaging unit to generate a visible light imaging signal under visible light imaging conditions in which the infrared cut filter is located at the blocking position, and that causes the imaging unit to generate an infrared mixed imaging signal under infrared mixed imaging conditions in)
As to claim 3, OTSU teaches the instructions when executed by the at least one processor further perform operations as a separating unit configured to separate the visible light image data and the nonvisible light image data from image data generated by the imaging unit ([0009], a separation unit that separates the infrared mixed imaging signal into a visible light calculation imaging signal and an infrared mixed imaging signal based on the ratio and/or difference of the signal amounts of the visible light imaging signal and the infrared mixed imaging signal).
As to claim 4, OTSU teaches the determining unit determines the imaging scene according to a setting by a user (claim 3, [0018], The imaging condition control unit 7 controls the infrared irradiating unit 1 and the filter driving unit 6 to switch between imaging conditions of visible light and infrared mixed imaging conditions and an operation unit that waits for controls witching from the infrared mixed imaging condition to the visible light imaging condition until an operation is performed by the operator).
As to claim 5, OTSU teaches the determining unit determines the imaging scene according to a result of object recognition using at least one of the visible light image data and the nonvisible light image data( [0061], FIG. 6 shows a case where gray world control is performed on an infrared mixed image signal RGBAxy obtained by capturing an image of the Japanese flag).
As to claim 8, OTSU teaches the determining unit determines the imaging scene using luminance information acquired from at least one of the visible light image data and the nonvisible light image data ([0037], For example, a first luminance signal Y1xy is generated by resampling the color-matched imaging signal c[RGBxy], and the infrared calculation imaging signal [Axy] is used as a second luminance signal Y2xy. These luminance signals Y1xy and Y2xy are then synthesized to generate the luminance signal Yxy).
As to claim 13, OTSU teaches in a case where the nonvisible light image data is generated and the image processing using the nonvisible light image data is executed ([0044], The operator sees the flashing and lit warning display and determines whether or not to execute control switching from the infrared mixed imaging condition to the visible light imaging condition.), the control unit records the nonvisible light image data in another memory according to a setting by a user([0074], imaging device includes a CPU, a main storage device such as a RAM, and a computer-readable storage medium in the system that stores a program for realizing all or part of the image processing. Thus, infrared image data and mixed imaging condition are the storage devise).
As to claim 14, OTSU teaches An image pickup apparatus comprising: an imaging unit; and the imaging control apparatus according to claim 1(Abstract).
As to claim 15, OTSU teaches a display unit configured to display to a user that the nonvisible light image data is generated and image processing using the nonvisible light image data is performed ([0044], The operator sees the flashing and lit warning display and determines whether or not to execute control switching from the infrared mixed imaging condition to the visible light imaging condition).
As to claim 17, OTSU teaches a display unit configured to display to a user that a nonvisible light source that irradiates nonvisible light onto an object is lit (([0044], The operator sees the flashing and lit warning display and determines whether or not to execute control switching from the infrared mixed imaging condition to the visible light imaging condition).
Claim 18 is rejected the same as claim 1 except claim 18 is directed to a method claim. All the limitations of claim 18 are addressed in claim 1. Thus, argument analogous to that presented above for claim 1 is applicable to claim 18.
As to claim 19, OTSU teaches A non-transitory computer-readable storage medium storing a program that causes a computer to execute the imaging control method according to claim 18([088], In this way, the imaging device according to an embodiment of the present invention may be realized by software processing, and if realized by software processing, changes or additions to the processing content can be easily accommodated by rewriting the program).
As to claim 20, the combination of OTSU and Wang teaches one or more of the following conditions is satisfied:
(4) the usage of image processing using the nonvisible light image data differs depending upon whether contrast improvement is being used (Wang , claim 12, wherein to generate the predicted sharp version of the input image is further to generate the predicted sharp version of the input image based on keeping a contrast distribution in the predicted sharp version consistent with a contrast distribution of a set of known image).
(5) the usage of image processing using the nonvisible light image data differs depending upon whether noise reduction is being used (OTSU: page 6 last two pars., the image processing unit 14 is not limited to the resampling process as described above, but various image processes such as luminance gradation correction, detail correction, various noise cancellation filters, color correction, and resolution conversion);
(6) the usage of image processing using the nonvisible light image data differs depending upon whether image stabilization is being used (Wang: Abstract col.4 lines 30-45, a method of estimation blur degradation of an image (for example it is known that blur can be caused by camera shake ), and correcting the blur degradation of the image by deriving a blur kernel; and deblurring the image by deconvolution using the derived blur kernel, where the camera employ visible or non-visible wavelengths such as infrared
frequencies).
8. Claims 6-7, 10, 12 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over OTSU, JP-2010154266A, in view of Wang, US 8897588 B2, further in view of YONISHI OSAMU ( hereafter YONISHI), KR 20190079574A, pub. 07/05/2019.
Regarding claim 6, while modified OTSU, teaches the limitation of claim 5, but fails to teach the limitation of claim 6.
On the other hand YONISHI teaches the determining unit selects image data to be used for the object recognition from the visible light image data and the nonvisible light image data according to the imaging scene (Page 7, 1st par.,- 3rd par., YONISHI specifically teaches a case when an object is detected in the infrared light image, it may be difficult to determine whether the infrared light image should be converted into a visible light image. In such a case, machine learning is applied to the object determining process, and the type of object is determined based on features such as shape and size. Thereafter, the infrared light image may be switched to the visible light image only when an object of a specific detection level or higher is specified. The "detection level" indicates the degree to which an object should be monitored).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a machine learning technique configured to detected object using visible light image or infrared image based on features such as shape size of the object to be detected and also based on the weather conditions taught by YONISHI into modified OTSU.
The suggestion/motivation for doing so would have been to allow user of modified OTSU to detected object more efficiently by utilizing learned visible light image data and leaned infrared image light which are generated by training a large number of image data using ML technique.
As to claim 7, YONISHI teaches the determining unit uses a learning model for the object recognition, and wherein the determining unit uses a visible light image learning model in a case where the image data for the object recognition is the visible light image data, and the determining unit uses a nonvisible light image learning model as the learning model in a case where the image data for the object recognition is the nonvisible light image data((Page 7, 2nd par.,- 3rd par., as discussed in claim 6 above YONISHI teaches a case when an object
is detected in the infrared light image, it may be difficult to determine whether the infrared light image should be converted into a visible light image. In such a case,
machine learning is applied to the object determining process, and the type of object is determined based on features such as shape and size. Thereafter, the infrared light image may be switched to the visible light image only when an object of a specific detection level or higher is specified. The "detection level" indicates the degree to which an object should be monitored).
As to claim 10, YONISHI teaches the determining unit determines the imaging scene using object distance information(page 6 par. 3, page 2 par., 2., the infrared camera is suitable for monitoring at a distance. In comparison with a visible light image, it is possible to detect an object at a long distance in an ultraviolet light image. Thus, a distance information is utilized when detecting an object using infrared camera).
As to claim 12, YONISHI teaches the control unit controls whether or not to emit a nonvisible light source that irradiates nonvisible light onto an object according to a determination result of the imaging scene ( Page 7, 1st to par.,- 3rd par., when the visible light image is always distributed, the object cannot be detected when the photographing condition is bad. According to the above-described embodiment, it is possible to provide an image suitable for monitoring difficult to be affected by weather condition. In this situation visible light image may be switched to the infrared light only when an object of a specific detection level or higher is specified).
As to claim 16, OTSU teaches display unit configured to prompt a user to execute image processing, visible image data and nonvisible light image data (this limitation discussed in claim 1 above);
however, it is noted that OTSU does not specifically under line section of the limitation of “a display unit configured to prompt a user to execute image processing using the nonvisible light image data in a case where generation of the nonvisible light image data and image processing using the nonvisible light image data is not performed “
On the other hand YONISHI teaches a display unit configured to prompt a user to execute image processing using the nonvisible light image data in a case where generation of the nonvisible light image data and image processing using the nonvisible light image data is not performed (Page 7, 1st- par.,- 3rd par., when the visible light image is always distributed, the object cannot be detected when the photographing condition is bad. According to the above-described embodiment, it is possible to provide an image suitable for monitoring difficult to be affected by weather condition. In this situation visible light image may be switched to the infrared light only when an object of a specific detection level or higher is specified).
9. Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over OTSU,
JP2010154266A, in view of Wang, US 8897588 B2, further in view of ZHANG et al., (hereafter, ZHANG), WO 2021217643 A, pub. 11/04/2021.
Regarding claim 9, while modified OTSU, teaches the limitation of claim 1, but fails to teach the limitation of claim 9.
On the other hand ZHANG the determining unit determines the imaging scene using motion vector information acquired from at least one of the visible light image data and the nonvisible light image data(page 7 4th par., For example, the relative position of a visible light sensor and the infrared sensor is fixed, and the two respectively collect visible images and infrared images in the same scene. Since the relative positions of the two are fixed, their global motion vectors are the same. Since the resolution of the visible light image is higher, the motion vector determined from the visible light image will be more accurate. Therefore, the motion vector of the visible light image can be combined to assist in determining the motion vector of the infrared image and its reference image, so that the determined motion vector is more accurate).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a method of generating a motion vector by combining visible light and infrared (IR) images taught by ZHANG into modified OTSU.
The suggestion/motivation for doing so would have been to allow user of modified OTSU to generate a robust object detection method by combining rich texture and detail of the object obtained from visible images data of the object, and thermal radiation data of the object obtained from the infrared image data of the object regardless of ambient light.
10. Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over OTSU,
JP-2010154266A, in view of Wang, US 8897588 B2,further in view of YONISHI, KR 20190079574A, further in view of IKEDA, JUN (hereafter IKEDA), WO 2010109919A, pub. 09/30/2010.
As to claim 11, YONISHI teaches the control unit controls whether or not to generate the nonvisible light image data and whether or not to execute image processing using the nonvisible light image data according to a determination result of the imaging scene (Page 7, 2nd par.,- 3rd par., as discussed in claim 6 above YONISHI teaches a case when an object is detected in the infrared light image, it may be difficult to determine whether the infrared light image should be converted into a visible light image. In such a case, machine learning is applied to the object determining process, and the type of object is determined based on features such as shape and size. Thereafter, the infrared light image may be switched to the visible light image only when an object of a specific detection level or higher is specified. The "detection level" indicates the degree to which an object should be monitored).
However, it is noted that the combination of OTSU and YONISHI does not specifically teaches the underline section of the limitation “to execute image processing using the nonvisible light image data according to a determination result of a shake detection result of an image pickup apparatus”.
On the other hand JUN teaches to execute image processing using the nonvisible light image data according to a determination result of a shake detection result of an image pickup apparatus (page 5 last paragraph and page 17 last two paragraphs, the imaging device 9 can detect the shake of the subject from the infrared video signal obtained by irradiating the subject with infrared light that allows the subject motion detection unit 23 to clearly shoot the subject even in a dark place such as at night. Further the camera shake can be detected from the captured infrared video signal, and the camera shake in the visible video signal is corrected based on the detected motion information of the subject)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to incorporate a method of detecting and correcting camera shake in the visible video signal by detect the shake of the subject from the infrared video signal taught JUN into modified OTSU.
The suggestion/motivation for doing so would have been to allow user of modified OTSU to generate visible images data by removing the artifact generated due to camera shake.
a robust object detection method by combining rich texture and detail of the object obtained.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Mekonen Bekele whose telephone number is (469) 295-9077.The examiner can normally be reached on Monday-Friday from 9:00AM to 6:50 PM Eastern Time.
If attempt to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Eng, George can be reached on (571) 272-7495.The fax phone number for the organization where the application or proceeding is assigned is 571-237-8300. Information regarding the status of an application may be obtained from the patent Application Information Retrieval (PAIR) system. Status information for published application may be obtained from either Private PAIR or Public PAIR. Status information for unpublished application is available through Privet PAIR only.
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have question on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866.
/MEKONEN T BEKELE/Primary Examiner, Art Unit 2699
Prosecution Insights