DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 2, 17, 37 and 38 are objected to because of the following informalities:
Line 2 of claim 2 recites “the depth information”, which should be changed to “depth information”. Appropriate correction is required.
Line 9 of claim 17 recites “the first image”, which should be changed to “a first image”. Line 10 of claim 17 recites “the second image”, which should be changed to “a second image”. Appropriate correction is required.
Line 5 of claim 37 recites “stud user”, which should be changed to “said user” or “the user”. Appropriate correction is required.
Line 6 of claim 38 recites “the first detector”, which should be changed to “the first sensor”. Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-5, 7, 9, 11, 12, 14, 16-21, 23, 25, 27, 30-35 and 38-42 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Lee et al. (US 2017/0339329 A1).
As to claim 1, Lee et al. discloses an image capturing device that captures images (Fig.1A; or Fig.5B: sensor device 510; or Fig.9), comprising:
a first sensor (Fig.5B: RGB sensor in sensor assembly 560(3); or Fig.9: image sensor of camera 970B) includes a first imaging modality (As shown in Fig.5B, the RGB sensor captures RGB color image information.) (The image sensor of camera 970B in Fig. 9 is also an RGB sensor. As disclosed in para [0058], the three different cameras 970A-C are similar to the three cameras 197A-C, except that all three cameras 970 are part of a single sensor assembly 960; therefore, the image sensor of camera 970B is an RGB sensor) (Fig.1A and para [0025] also discloses “Examples of sensor devices include cameras and other image capture devices, including monochrome, single-color, multi-color, RGB, other visible, IR, 4-color (e.g., RGB+IR), stereo, multi-view, strobed, and high-speed”);
a second sensor (Fig.5B: RGB sensor (Fisheye) in sensor assembly 560(3); or Fig.9: image sensor of camera 970C) includes the first imaging modality (RGB sensor (Fisheye) in Fig.5B captures RGB color image information.) (The image sensor of camera 970C in Fig.9 is a high-resolution RGB sensor; therefore, it also captures RGB image information);
a third sensor (Fig.5B: TOF depth sensor in sensor assembly 560(3); or Fig.9: image sensor of camera 970A) includes a second imaging modality (TOF depth sensor captures depth information.) (The image sensor of camera 970A in Fig.9 is similar to image sensor 197A in Fig.1B; it is a RGB+IR sensor; therefore, it captures RGB and IR information, which corresponds to the second imaging modality in the claim); and
a controller (Fig.5B: Application processor 525; or Fig.9 and [0058]: the application processor within the sensor base 920) connected with the first sensor, the second sensor and the third sensor (As shown in Fig.5B, the sensor assemblies 560 are connected to the application processor 525 in the sensor base 520. Fig.9 also shows that sensor assembly 960 is connected to sensor base 920), wherein the controller registers an image captured by the first sensor or the second sensor to an image captured by the third sensor ([0037]: “…combination of different images or with other data from different sensor channels.” “…imagery may be registered to depth information to produce an image with depth or even a three-dimensional model”).
As to claim 2, Lee et al. discloses the image capturing device of claim 1, wherein the image registration is based on the depth information calculated based on information from the first sensor and the second sensor (Fig.4A; [0043]: “The Multi-focal CaptureMode captures images using different focal lengths. From this, depth estimates can be produced”. [0037]: “imagery may be registered to depth information to produce an image with depth or even a three-dimensional model”).
As to claim 3, Lee et al. discloses the image capturing device of claim 1, wherein the first imaging modality comprises color imaging (Fig.5B: RGB sensor).
As to claim 4, Lee et al. discloses the image capturing device of claim 1, wherein the first imaging modality comprises near-infrared imaging (Fig.1A; [0025]: “Examples of sensor devices include cameras and other image capture devices, including monochrome, single-color, multi-color, RGB, other visible, IR, 4-color (e.g., RGB+IR), stereo, multi-view, strobed, and high-speed”. Therefore, in a case where the first sensor and the second sensor are IR sensors, the first imaging modality is IR/near-infrared imaging).
As to claim 5, Lee et al. discloses the image capturing device of claim 1, wherein the second imaging modality comprises thermal imaging (Fig.1A; [0025]: “Examples of sensor devices include cameras and other image capture devices, including monochrome, single-color, multi-color, RGB, other visible, IR, 4-color (e.g., RGB+IR), stereo, multi-view, strobed, and high-speed”. The IR sensor corresponds to the claimed thermal imaging).
As to claim 7, Lee et al. discloses the image capturing device of claim 1, further comprising a light source to illuminate a subject captured by at least one of the first sensor, the second sensor and the third sensor ([0025]: Structured light devices; [0055]: IR/visible flash).
As to claim 9, Lee et al. discloses the image capturing device of claim 1, further comprising at least one actuator connected with at least one of the first sensor, the second sensor and the third sensor, for adjusting a position of at least one of the first sensor, the second sensor and the third sensor ([0054]: the setting passed across the ISM interface may include zoom, pan and tilt, and focus; therefore, an actuator/motor is inherently included, in order to adjust the zoom, pan and tilt, and focus of the sensor).
As to claim 11, Lee et al. discloses the image capturing device of claim 1, wherein the first sensor, the second sensor and the third sensor provide information to control a vehicle ([0003]: “Applications such as advanced driver assisted systems and security based on facial recognition require custom built software which reads in raw images from cameras and then processes the raw images in a specific way for the target application”. The camera system is considered that at least partially controls a vehicle when it is used in advanced driver assisted system).
As to claim 12, Lee et al. discloses the image capturing device of claim 1, further comprising a display (Fig.3: for example, when the system is used in a home surveillance system, a display is inherently included in the system in order to monitor the scene and play back the recorded video) to display an output of the registered image ([0037]: “Examples of ProcessedData 222 include filtered and enhanced images, and the combination of different images or with other data from different sensor channels”. The combination of different images corresponds the registered image in the claim).
As to claim 14, Lee et al. discloses the image capturing device of claim 1, further comprising a commutation interface to remotely communicate data from the first sensor, the second sensor and the third sensor (Fig.1B; [0030]: “the ISM 195 is an interchangeable connectorless camera mount”. Other examples are shown in Fig.5B, ISMs 550(1)-550(3)).
As to claim 16, Lee et al. discloses the image capturing device of claim 1, further comprising a filter positioned in front of at least one of the first sensor, the second sensor and the third sensor and a subject (As shown in Figs.1B and 5B, some of the sensors are RGB sensors, which contain RGB color filters).
As to claim 17, Lee et al. an image capturing device (Fig.5B: sensor assembly 560(3); or Fig.9: sensor assembly 960) for use with a device (Fig.5B: sensor base 520; or Fig.9: sensor base 920) having a controller (Fig. 5B: application processor 525; or the application processor in the sensor base 920 in Fig.9) and a communication interface (Fig.5B: ISM 550(3); [0030]: “the ISM 195 is an interchangeable connectorless camera mount”; or Fig.9 and [0058]: ISM 950), comprising:
a first sensor (Fig.5B: RGB sensor in sensor assembly 560(3); or Fig.9: image sensor of camera 970B) includes a first imaging modality (As shown in Fig.5B, the RGB sensor captures RGB color image information.) (The image sensor of camera 970B in Fig. 9 is also an RGB sensor. As disclosed in para [0058], the three different cameras 970A-C are similar to the three cameras 197A-C, except that all three cameras 970 are part of a single sensor assembly 960; therefore, the image sensor of camera 970B is an RGB sensor) (Fig.1A and para [0025] also discloses “Examples of sensor devices include cameras and other image capture devices, including monochrome, single-color, multi-color, RGB, other visible, IR, 4-color (e.g., RGB+IR), stereo, multi-view, strobed, and high-speed”);
a second sensor (Fig.5B: RGB sensor (Fisheye) in sensor assembly 560(3); or Fig.9: image sensor of camera 970C) includes the first imaging modality (RGB sensor (Fisheye) in Fig.5B captures RGB color image information.) (The image sensor of camera 970C in Fig.9 is a high-resolution RGB sensor; therefore, it also captures RGB image information);
a third sensor (Fig.5B: TOF depth sensor in sensor assembly 560(3); or Fig.9: image sensor of camera 970A) includes a second imaging modality (TOF depth sensor captures depth information.) (The image sensor of camera 970A in Fig.9 is similar to image sensor 197A in Fig.1B; it is a RGB+IR sensor; therefore, it captures RGB and IR information, which corresponds to the second imaging modality in the claim); and
a communication interface (Fig.5B: ISM 550(3); [0030]: “the ISM 195 is an interchangeable connectorless camera mount”. Please note that both of the sensor based 520 and sensor assembly 560 include a communication interface in order to communicate with each other) (Or Fig.9: ISM 950) adapted to communicate with the communication interface of the device to send an image captured by at least one of the first sensor and the second sensor, and an image captured by the third sensor, to the controller (Fig.2; [0036]: the data captured by the sensor devices 210 is transmitted to the application processor 220) (Fig.9; [0058]: “The solid black arrows indicate the flow of captured image data”);
wherein the controller of the device registers the first image captured by the first sensor to the second image captured by the third sensor (Fig.4A; [0043]: “images of a Scene are captured by multiple cameras and these images are synchronized and registered to each other”).
As to claim 18, Lee et al. discloses the image capturing device of claim 17, wherein the registration is based on the depth information calculated based on the first sensor and the second sensor (Fig.4A; [0043]: “The Multi-focal CaptureMode captures images using different focal lengths. From this, depth estimates can be produced”. [0037]: “imagery may be registered to depth information to produce an image with depth or even a three-dimensional model”).
As to claim 19, Lee et al. discloses the image capturing device of claim 17, wherein the first imaging modality comprises color imaging (Fig.5B: RGB sensor).
As to claim 20, Lee et al. discloses the image capturing device of claim 17, wherein the first imaging modality comprises near-infrared imaging (Fig.1A; [0025]: “Examples of sensor devices include cameras and other image capture devices, including monochrome, single-color, multi-color, RGB, other visible, IR, 4-color (e.g., RGB+IR), stereo, multi-view, strobed, and high-speed”. Therefore, in a case where the first sensor and the second sensor are IR sensors, the first imaging modality is IR/near-infrared imaging).
As to claim 21, Lee et al. discloses the image capturing device of claim 17, wherein the second imaging modality comprises thermal imaging (Fig.1A; [0025]: “Examples of sensor devices include cameras and other image capture devices, including monochrome, single-color, multi-color, RGB, other visible, IR, 4-color (e.g., RGB+IR), stereo, multi-view, strobed, and high-speed”. The IR sensor corresponds to the claimed thermal imaging).
As to claim 23, Lee et al. discloses the image capturing device of claim 17, further comprising a light source to illuminate a subject captured by at least one of the first sensor, the second sensor and the third sensor ([0025]: Structured light devices; [0055]: IR/visible flash).
As to claim 25, Lee et al. discloses the image capturing device of claim 17, further comprising at least one actuator connected with at least one of the first sensor, the second sensor and the third sensor, for adjusting a position of at least one of the first sensor, the second sensor and the third sensor ([0054]: the setting passed across the ISM interface may include zoom, pan and tilt, and focus; therefore, an actuator/motor is inherently included, in order to adjust the zoom, pan and tilt, and focus of the sensor).
Claim 27 recites substantially similar subject matter as disclosed in claim 12; therefore, it is rejected for the same reasons.
As to claim 30, Lee et al. discloses the image capturing device of claim 17, further comprising a filter positioned in front of at least one of the first sensor, the second sensor and the third sensor and a subject (As shown in Figs.1B and 5B, some of the sensors are RGB sensors, which contain RGB color filters).
As to claim 31, Lee et al. discloses a method of registering images of a first imaging modality to images of a second imaging modality, comprising:
capturing a first image using a first sensor (Fig.9: image sensor of camera 970B) that includes a first imaging modality (The image sensor of camera 970B in Fig. 9 is an RGB sensor. As disclosed in para [0058], the three different cameras 970A-C are similar to the three cameras 197A-C, except that all three cameras 970 are part of a single sensor assembly 960; therefore, the image sensor of camera 970B is an RGB sensor) (Fig.1A and para [0025] also discloses “Examples of sensor devices include cameras and other image capture devices, including monochrome, single-color, multi-color, RGB, other visible, IR, 4-color (e.g., RGB+IR), stereo, multi-view, strobed, and high-speed”);
capturing a second image using a second sensor (Fig.9: image sensor of camera 970C) that includes the first imaging modality (The image sensor of camera 970C in Fig.9 is a high-resolution RGB sensor; therefore, it also captures RGB image information);
capturing a third image using a third sensor (Fig.9: image sensor of camera 970A) that includes a second imaging modality (The image sensor of camera 970A in Fig.9 is similar to image sensor 197A in Fig.1B; it is a RGB+IR sensor; therefore, it captures RGB and IR information, which corresponds to the second imaging modality in the claim);
determining a first depth map for at least one pixel of the first image based on the first image and the second image ([0043]: “The Multi-focal CaptureMode captures images using different focal lengths. From this, depth estimates can be produced”. The depth information corresponds to the depth map in the claim); and
registering the first image or the second image to the third image based on the first depth map ([0037]: “imagery may be registered to depth information to produce an image with depth or even a three-dimensional mode”).
As to claim 32, Lee et al. discloses the method of claim 31, further comprising displaying a composite image of the first image registered to the third image ([0037]: “Examples of ProcessedData 222 include filtered and enhanced images, and the combination of different images or with other data from different sensor channels”).
As to claim 33, Lee et al. discloses the method of claim 31, wherein the first imaging modality comprises color imaging (Please note that the cameras 970A-C in Fig.9 is similar to the cameras 197A-C in Fig.1B. Therefore, 970B is similar to camera 197B, which contains RGB sensor).
As to claim 34, Lee et al. discloses the method of claim 31, wherein the first imaging modality comprises near-infrared imaging (Fig.1A; [0025]: “Examples of sensor devices include cameras and other image capture devices, including monochrome, single-color, multi-color, RGB, other visible, IR, 4-color (e.g., RGB+IR), stereo, multi-view, strobed, and high-speed”. Therefore, in a case where the first sensor and the second sensor are IR sensors, the first imaging modality is IR/near-infrared imaging).
As to claim 35, Lee et al. discloses the method of claim 31, wherein the second imaging modality comprises at least of one of thermal imaging, terahertz imaging, hyperspectral imaging, fluorescence imaging, narrow band imaging and oxygen saturation imaging (Fig.1A; [0025]: “Examples of sensor devices include cameras and other image capture devices, including monochrome, single-color, multi-color, RGB, other visible, IR, 4-color (e.g., RGB+IR), stereo, multi-view, strobed, and high-speed”. The IR sensor corresponds to the claimed thermal imaging).
As to claim 38, Lee et al. discloses an image capturing device that captures images (Fig.1A; or Fig.5B: sensor device 510; or Fig.9), comprising:
a first sensor (Fig.5B: RGB sensor (Fisheye) in sensor assembly 560(3); or Fig.9: image sensor of camera 970C) includes a first imaging modality (RGB sensor (Fisheye) in Fig.5B captures RGB color image information.) (As disclosed in para [0058], the three different cameras 970A-C are similar to the three cameras 197A-C, except that all three cameras 970 are part of a single sensor assembly 960; therefore, the image sensor of camera 970C is similar to 197C in Fig.1B, which contains RGB sensor. Therefore, it also captures RGB image information.) (Fig.1A and para [0025] also discloses “Examples of sensor devices include cameras and other image capture devices, including monochrome, single-color, multi-color, RGB, other visible, IR, 4-color (e.g., RGB+IR), stereo, multi-view, strobed, and high-speed”);
a second sensor (Fig.5B: RGB sensor in sensor assembly 560(3); or Fig.9: image sensor of camera 970B) includes the second imaging modality (As shown in Fig.5B, the RGB sensor captures RGB color image information.) (The image sensor of camera 970B in Fig. 9 is similar to camera 197B in Fig.1B; therefore, it is also an RGB sensor);
an actuator ([0054]: the setting passed across the ISM interface may include zoom, pan and tilt, and focus; therefore, an actuator/motor is inherently included, in order to adjust the zoom, pan and tilt, and focus of the sensor); and
a controller (Fig.5B: Application processor 525; or Fig.9 and [0058]: the application processor within the sensor base 920) connected with the first sensor, the second sensor and the actuator (As shown in Fig.5B, the sensor assemblies 560 are connected to the application processor 525 in the sensor base 520. Fig.9 also shows that sensor assembly 960 is connected to sensor base 920), wherein the controller, responsive to a request from the first detector, adjusts the position of the actuator to the requested position to capture an image by the second sensor ([0056]: “The parameters may also specify the coordination of different types of sensors, for example the coordinated capture of right and left stereo pairs”. That is, when capturing the stereo images, the right and left cameras have to coordinate with each other in order to have a similar field of view to capture the same scene).
As to claim 39, Lee et al. discloses the image capturing device of claim 38, wherein the first sensor further comprises at least one of an omnidirectional lens, a 360 degree lens, a fisheye lens, an wide-angle lens and a convex mirror (Fig.5B: RGB sensor (Fisheye)).
As to claim 40, Lee et al. discloses the image capturing device of claim 38, wherein the first sensor further comprises a thermal image sensor (Fig.1A; [0025]: “Examples of sensor devices include cameras and other image capture devices, including monochrome, single-color, multi-color, RGB, other visible, IR, 4-color (e.g., RGB+IR), stereo, multi-view, strobed, and high-speed”. The IR sensor corresponds to the claimed thermal imaging).
As to claim 41, Lee et al. discloses the image capturing device of claim 38, wherein the first sensor further comprises a color image sensor having a large field of view and the second sensor further comprises a color image sensor having a small field of view ([0048]: “Sensor assembly 560(1) provides two additional color cameras with wide angle and telephoto capabilities”).
As to claim 42, Lee et al. discloses the image capturing device of claim 38, wherein information from the first sensor and the second sensor control at least one of a wheel, an unmanned aerial vehicle, a display, a robot, a toy, a baby monitor and a head-mounted display.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 6, 8, 22 and 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US 2017/0339329 A1) in view of Liu (US 2018/0270474 A1).
As to claim 8, Lee et al. discloses the image capturing device of claim 7. It fails to disclose the light source further comprises at least one light emitting diode.
However, Liu teaches the light source further comprises at least one light emitting diode ([0112]: light source 450 may comprise a light emitting diode (LED)).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Lee et al. with the teaching of Liu to use at least one light emitting diode, so as to provide a light source that is durable and energy efficient.
As to claim 6, Lee et al. discloses the image capturing device of claim 1 but fails to disclose the second imaging modality comprises at least of one of terahertz imaging, hyperspectral imaging, fluorescence imaging, narrow band imaging and oxygen saturation imaging.
However, Liu teaches the second imaging modality comprises at least of one of terahertz imaging, hyperspectral imaging, fluorescence imaging, narrow band imaging and oxygen saturation imaging ([0104-0107] discloses hyperspectral imaging, fluorescence imaging and oxygen saturation imaging).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Lee et al. with the teaching of Liu to include at least one of terahertz imaging, hyperspectral imaging, fluorescence imaging, narrow band imaging and oxygen saturation imaging as the second imaging modality, so as to enhance the definition and sharpness of the image.
Claim 22 recites substantially similar subject matter as in claim 6; therefore, it is rejected for the same reasons.
Claim 24 recites substantially similar subject matter as disclosed in claim 8; therefore, it is rejected for the same reasons.
Claims 10, 13, 26, 28 and 42 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US 2017/0339329 A1) in view of Kim et al. (US 2017/0085755 A1, cited by applicant).
As to claims 10 and 13, Lee et al. discloses the image capturing device of claim 1. It fails to disclose an unmanned aerial vehicle; it also fails to disclose the first sensor, the second sensor and the third sensor are housed in at least one of a robot, a motor vehicle, a toy, a baby monitor and a head-mounted display.
However, Kim et al. teaches that the camera system is applicable to an unmanned aerial vehicle ([0177]) and a vehicle ([0174]).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Lee et al. with the teaching of Kim et al. to use the camera system in an unmanned aerial vehicle, and house the first sensor, the second sensor and the third sensor in a motor vehicle, so as to capture images from different view point by using unmanned aerial vehicle equipped with camera system, and provide parking assistance system, car safety assistant system, advanced driver assistant system, etc., by incorporating the camera system in a vehicle, thereby expand the applications of the camera system.
Claims 26 and 42 recite substantially similar subject matter as disclosed in claim 10; therefore, they are rejected for the same reasons.
Claims 28 recites substantially similar subject matter as disclosed in claim 13; therefore, it is rejected for the same reasons.
Claims 15 and 29 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US 2017/0339329 A1) in view of Rouh et al. (US 2016/0094833 A1, cited by applicant).
As to claim 15, Lee et al. discloses the image capturing device of claim 1; it fails to disclose the third sensor having the second imaging modality is positioned between or on top of the first sensor and the second sensor having the first imaging modality.
However, Rouh et al. teaches a third camera is positioned on top of the first camera and the second camera (Fig.1; [0068]).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Lee et al. with the teaching of Rouh et al. to position the third sensor having the second imaging modality on top of the first sensor and the second sensor having the first imaging modality, so as to obtain good triangulation accuracy and determine the position of subject of interest ([0015]).
Claim 29 recites substantially similar subject matter as disclosed in claim 15; therefore, it is rejected for the same reasons.
Claims 36 and 37 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US 2017/0339329 A1) in view of Kobayashi et al. (US 2013/0249904 A1).
As to claim 36, Lee et al. discloses the method of claim 31. While Lee et al. discloses registering images based on depth information/map ([0037]: “…combination of different images or with other data from different sensor channels.” “imagery may be registered to depth information to produce an image with depth or even a three-dimensional model”), it fails to disclose calculating a second depth map for at least one pixel of the second image based on the first image and the second image; and registering the second image to the third image based on the second depth map.
However, Kobayashi et al. teaches calculating a second depth map for at least one pixel of the second image based on the first image and the second image (Fig.4; [0103-0104]: the depth detecting unit 203 generates left image depth information and right image depth information in block increments or pixel increments. One of the left image depth information and right image depth information may correspond to the claimed second depth map).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Lee et al. with the teaching of Kobayashi et al. to calculate a second depth map for at least one pixel of the second image based on the first image and the second image; and register the second image to the third image based on the second depth map, so as to generate more accurate depth map for the second image, thereby generating the registered image with better quality.
As to claim 37, Lee et al. in view Kobayashi et al. discloses the method of claim 36, further comprising: displaying a first composite image of the first image registered to the third image to a first eye of a user; and displaying a second composite image of the second image registered to the third image to a second eye of stud user ([0050]: the left image is for left eye; and the right image is for right eye).
Claims 43-46 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US 2017/0339329 A1) in view of Hyun (US 2014/0320664 A1, cited by applicant).
As to claim 43, Lee et al. discloses an image capturing device (Fig.5A discloses sensor base 520 and sensor assembly 560 are connected to each other; while Fig.5B shows that sensor base 520 can connect to multiple sensor assemblies 560(1)-560(3), which may be stacked or daisy chained. Therefore, sensor base 520 and sensor assembly 560(2) correspond to the claimed image capturing device) for use with a device (the sensor assembly 560 in Fig.5A, or sensor assembly 560(3), may correspond to the claimed device) having an image sensor (Fig.5B: TOF depth sensor) including a second imaging modality (TOF sensor captures depth information, which corresponds to the second imaging modality), a controller (Fig.5A: controller 565) and a communication interface (Fig.5A: interface 552(B)), comprising:
a first sensor (Fig.5B: RGB sensor in sensor assembly 560(3)) includes a first imaging modality (As shown in Fig.5B, the RGB sensor captures RGB color image information.);
an actuator, the actuator mechanically coupled to the device ([0054]: the setting passed across the ISM interface may include zoom, pan and tilt, and focus; therefore, an actuator/motor is inherently included, in order to adjust the zoom, pan and tilt, and focus of the sensor); and
a communication interface (Fig.5A: ISMs 550(A) and 550 (B); Fig.5B: ISMs 550(1)-550(3)) adapted to communicate with the communication interface of the device to send an image captured by first sensor to the controller ([0030]: “It is interchangeable in the sense that different camera assemblies 197A-C may be attached/detached to the mount 195 and the mount 195 also provides an interface to allow the different assembles 197 to interoperate with the sensor base 198).
Lee et al. fails to disclose the controller, responsive to a request from the first sensor, adjusts a position of the actuator to a requested position to capture an image by the image sensor of the device.
However, Hyun teaches the controller (Fig. 2: control unit 220), responsive to a request from the first sensor (Fig.3; [0077]: step S4 – select subject when surveillance object selection signal is received as tracking surveillance subject) (Fig.2: the first cameras 111 and 121 correspond to the first sensor), adjusts a position of the actuator to a requested position to capture an image by the image sensor of the device (Fig.3; [0078]: step S5 – photograph tracking surveillance subject using second camera. An actuator is inherently included in order to move the camera for object tracking).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify Lee et al. with the teaching of Hyun to have the controller, responsive to a request from the first sensor, adjust a position of the actuator to a requested position to capture an image by the image sensor of the device, so as to provide an imaging system having effective performance and enhanced functionality.
As to claim 44, Lee et al. in view of Hyun discloses the image capturing device of claim 43, wherein the first sensor further comprises at least one of an omnidirectional lens, a 360 lens, a fisheye lens, a wide-angle lens and a convex mirror (Lee et al.: Fig.5B: RGB sensor (Fisheye)).
As to claim 45, Lee et al. in view of Hyun discloses the image capturing device of claim 43, wherein the first sensor further comprises a thermal image sensor (Lee et al.: Fig.1A; [0025]: “Examples of sensor devices include cameras and other image capture devices, including monochrome, single-color, multi-color, RGB, other visible, IR, 4-color (e.g., RGB+IR), stereo, multi-view, strobed, and high-speed”. The IR sensor corresponds to the claimed thermal imaging).
As to claim 46, Lee et al. in view of Hyun discloses the image capturing device of claim 43, wherein the first sensor further comprises a color image sensor having a large field of view and the second sensor further comprises a color image sensor having a small field of view (Lee et al.: [0048]: “Sensor assembly 560(1) provides two additional color cameras with wide angle and telephoto capabilities”).
Claim 47 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lee et al. (US 2017/0339329 A1) in view of Hyun (US 2014/0320664 A1, cited by applicant) as applied to claim 43 above, and further in view of Kim et al. (US 2017/0085755 A1, cited by applicant).
As to claim 47, Lee et al. in view of Hyun discloses the image capturing device of claim 43. It fails to disclose information from the first sensor and the second sensor control at least one of a wheel, an unmanned aerial vehicle, a display, a robot, a toy, a baby monitor and a head-mounted display.
However, Kim et al. teaches that the camera system is applicable to an unmanned aerial vehicle ([0175-0176] discloses that the camera system can be used in vehicle to provide parking assistance system, car safety assistant system, advanced driver assistant system, etc. On the other hand, para [0177] discloses that the camera system is applicable to not only the vehicle but also any type of a system using a plurality of cameras such as unmanned aerial vehicle).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to modify the combination of Lee et al. and Hyun with the teaching of Kim et al. to use information from the first sensor and the second sensor to control an unmanned aerial vehicle, so as to provide safety assistant system and advanced driving assistant system for an unmanned aerial vehicle.
wherein information from the first sensor and the second sensor control at least one of a wheel, an unmanned aerial vehicle, a display, a robot, a toy, a baby monitor and a head-mounted display.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHENZHEN WU whose telephone number is (571)272-2519. The examiner can normally be reached 8:30 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, SINH TRAN can be reached at (571)272-7564. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZHENZHEN WU/Examiner, Art Unit 2637
/SINH TRAN/Supervisory Patent Examiner, Art Unit 2637
