Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed on 4-20-26 has been entered and fully considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, and 4-18 are rejected under 35 U.S.C. 103 as being unpatentable over Simek et al. (US 2018/0139431) in view of Spears et al. (US 2018/0324399).
Regarding claim 1, Simek (Fig. 2) discloses an imaging apparatus comprising:
a camera (206) to capture an image of an object in a space (“cameras 206 configured to capture 2D image data” discussed in [0067]);
a projector (as part of 204, “each depth detection components 204 can include… infrared light projectors” discussed in [0080]) to project light to the space (eg. infrared light);
a light receiver (as part of 204, “each depth detection components 204 can include a pair of infrared stereo cameras” discussed in [0080]) to receive light reflected from the object (eg. “light emitters and light receivers to capture depth information (e.g. time-of-flight sensor devices…” discussed in [0080], where the time-of-flight is a measurement of the time between emitting light, and detecting the light after being reflected off an object); and
a camera housing (202) to which the camera, the projector, and the light receiver are attached (as seen in Fig. 2, and as discussed above, 202 houses 204 and 206), wherein the plurality of imaging optical elements, the plurality of projecting optical elements, and the plurality of light receiving optical elements are at different positions in a direction of the camera housing (as discussed above and seen in Fig. 2, for example “arranged at different positions on the housing 202 and having different azimuth orientations relative to a center point (e.g. point 203)” discussed in [0067]);
a display (108); and
circuitry (514) configured to output, to the display, two-dimensional image information captured by the camera (“display 108 at which the raw and/or processed 2D images… can be presented” discussed in [0058], with “cameras 206 configured to capture 2D image data” discussed in [0067]),
wherein the circuitry outputs three-dimensional information (“display 108 at which the raw and/or processed… 3D data can be presented” discussed in [0058]), the three-dimensional information being based on an output from the light receiver (“determine 3D data from one or more structured light sensor devices” discussed in [0101]).
However, Simek only teaches that “the user device 106 can be configured to render (e.g. via display 108) a panoramic 2D image as well as more advanced panoramic data such a panoramic color video, a panoramic 3D depth image, a panoramic 3D depth video, and/or 3D model/mesh, as it is generated during the capture process (e.g. via primary processing component 104, secondary processing component 110, and/or tertiary processing component 114) in real-time or substantially real-real time” (see [0058]), but fails to specifically teach or suggest that the 2D image is generated before the 3D image, and so fails to teach or suggest the claimed “circuitry outputs the two-dimensional image information before outputting three-dimensional information.”
Spears (Fig. 1, 7, and 9) discloses an imaging apparatus comprising:
a camera (742) to capture an image of an object in a space (“capturing 2D image 704a by camera sensor 742a” discussed in [0057]);
a projector (124) to project light to the space (“124 projects light” discussed in [0029]);
a light receiver (126) to receive light reflected from the object (“for detecting light and/or reflection of light projected by an emitter (e.g., emitter module 124)” discussed in [0029]); and
a camera housing (100) to which the camera, the projector, and the light receiver are attached (as seen in Fig. 7), wherein the plurality of imaging optical elements, the plurality of projecting optical elements, and the plurality of light receiving optical elements are at different positions in a direction of the camera housing (seen in Fig. 7, see also the similar “camera sensor 142, emitter module 124, and receiver module 126 are configured on a same side of image capturing device 100. In one example, camera sensor 142, emitter module 124, and receiver module 126 may be aligned along a same baseline/axis” discussed in [0044]);
a display (145); and
circuitry (104) configured to output to the display (“GUI can be rendered by CPU(s) 104 for viewing on display 145” discussed in [0028]), and two-dimensional image information captured by the camera (“camera sensor 142 is used to capture two-dimensional (2D) image 204” discussed in [0032]),
wherein the circuitry outputs three-dimensional information (“3D image 210 may be provided to… at least one output device (e.g., display 145)” discussed in [0032]), the three-dimensional information being based on an output from the light receiver (eg. by using information from 126 to “calculate the depth of objects in the scene” as discussed in [0045], and then “generates a first wide-angle 3D image (wide-angle 3D image 710a) by aligning/merging first depth map 708a and 2D image 704a” as discussed in [0063]).
Spears additionally discloses wherein the circuitry generates the two-dimensional image information before generating three-dimensional information (as seen in Fig. 9, in 908 and 910 the 2D image is generated first, see “camera sensor 742a captures 2D image 704a within a first portion of scene 202. At block 910, camera sensor 742n contemporaneously captures 2D image 704n within a second portion of scene 202” discussed in [0062], and then only afterwards in 918 “104 generates a first wide-angle 3D image” as discussed in [0063]).
Therefore, the combination of Simek and Spears would provide circuitry which outputs the two-dimensional image information before outputting three-dimensional information (Simek teaches that each of the 2D and 3D images are displayed “as it is generated” in “real time” in [0058] while Spears teaches that the 2D images are generated before the 3D image, see [0063] and Fig. 9).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Simek so circuitry outputs the two-dimensional image information before outputting three-dimensional information as taught by Spears because both Simek and Spears teach that images can be displayed to a user (eg. to provide the user with “visual feedback during the capture process” as discussed in [0058] of Simek, and “3D image 210 may be provided to… at least one output device (e.g., display 145)” discussed in [0032] of Spears) and this allows the 3D images to be based on the 2D images (eg. the 3D image generated “by aligning/merging second depth map 708n and 2D image 704n” as discussed in [0063] of Spears).
Regarding claim 2, Simek and Spears disclose an imaging apparatus as discussed above, and Spears further discloses wherein the direction is parallel to a face of the camera housing (as seen in Fig. 7, the direction is the parallel to the side face of the housing), or the direction is a longitudinal direction of the camera housing (as seen in Fig. 7, the direction is the vertical longitudinal direction of the housing).
It would have been obvious to one of ordinary skill in the art to combine Simek and Spears for the same reasons as discussed above.
Regarding claim 4, Simek and Spears disclose an imaging apparatus as discussed above, and Spears further discloses wherein the plurality of imaging optical elements are at a same position in the direction (eg. 742a and 742n are aligned horizontally but on opposite sides of 100, as seen in Fig. 7), the plurality of projecting optical elements are at a same position in the direction (similarly, 124a and 124n are aligned horizontally in Fig. 7), and the plurality of light receiving optical elements are at a same position in the direction (similarly, 126a and 126n are aligned horizontally in Fig. 7).
It would have been obvious to one of ordinary skill in the art to combine Simek and Spears for the same reasons as discussed above.
Regarding claim 5, Simek and Spears disclose an imaging apparatus as discussed above, and Simek further discloses wherein the camera captures an image of a space including a measurement range (eg. the camera 206 captures 2D images of the environment corresponding to the measurement range of 204, see [0075] which discloses how 204 is “configured to capture and/or determine depth or distance information for features present in an environment, and more particularly visual features included in captured 2D images of the environment”), and the light receiver receives the light in a range including the measurement range (eg. “the range of the one or more depth sensor devices is up to about 6.0 meters” discussed in [0077]).
Regarding claim 6, Simek and Spears disclose an imaging apparatus as discussed above, and Simek further discloses wherein the camera captures an image of a full-view spherical space (as seen in Fig. 2B, see also “collective fields-of-view of the cameras 206… can span up to 360° relative to the horizontal quadrant plane” and 206 “can span about 180° in the vertical direction” discussed in [0069], which combines for a full spherical field of view).
Regarding claim 7, Simek and Spears disclose an imaging apparatus as discussed above, and Simek further discloses wherein the light receiver receives the light in a full-view spherical space (as seen in Fig. 2B, see also “collective fields-of-view of the cameras 206 and depth detection components 204, respectively, can span up to 360° relative to the horizontal quadrant plane” and “204 can span about 180° in the vertical direction” discussed in [0069], which combines for a full spherical field of view).
Regarding claim 8, Simek and Spears disclose an imaging apparatus as discussed above, and Spears further discloses wherein one of the plurality of imaging optical elements, one of the plurality of projecting optical elements, and one of the plurality of light receiving optical elements are lined in the direction (seen in Fig. 7, 742, 124, and 126 are aligned vertically on each side of 100, see also the similar “camera sensor 142, emitter module 124, and receiver module 126 are configured on a same side of image capturing device 100. In one example, camera sensor 142, emitter module 124, and receiver module 126 may be aligned along a same baseline/axis” discussed in [0044]).
It would have been obvious to one of ordinary skill in the art to combine Simek and Spears for the same reasons as discussed above.
Regarding claim 9, Simek and Spears disclose an imaging apparatus as discussed above, and Simek further discloses wherein an other camera (another of the cameras 206, with multiple cameras seen in Fig. 2A and “plurality of cameras 206” discussed in [0067]) to capture an image of the object, the other camera being on a same base line as the camera (eg. along the same horizontal plane of the housing, see “one located in each of the four corners of the housing 202” discussed in [0067]).
Regarding claim 10, Simek and Spears disclose an imaging apparatus as discussed above, and Simek further discloses wherein the circuitry (514) configured to instruct the display to display differently (eg. to remove the object from display, as discussed in [0107], or to not remove the object if it is absent, as discussed in [0108]) in accordance with presence or absences of a specific object (eg. based on the presence of the “camera operator” discussed in [0107]) in the image captured by the camera (determining whether to “subtract the object from the combined image data” is based after 514 is used to “analyze the respective 2D image captures” and “the color of the farthest 3D reading for each corresponding pixel” to identify “an object that was temporarily included in the space” as discussed in [0107]).
Regarding claim 11, Simek and Spears disclose an imaging apparatus as discussed above, and Simek further discloses wherein the circuitry (514) is configured to determine presence or absence of a specific object in the captured image (eg. determining the presence of the “camera operator” discussed in [0107]) based on an output from the light receiver and an output from the camera (514 is used to “analyze the respective 2D image captures” and “the color of the farthest 3D reading for each corresponding pixel” to identify “an object that was temporarily included in the space” as discussed in [0107]).
Regarding claim 13, Simek and Spears disclose an imaging apparatus as discussed above, and the combination further discloses wherein the circuitry outputs, to the display, the two-dimensional image information captured by the camera (as taught by Simek, “communication component 402 can be configured to facilitate wired and/or wireless communication between the 2D/3D panoramic capture device 400 and an external device, such as the user device 106” as discussed in [0089], while the user device 106 includes display 108 for displaying the 2D information, see [0058] which discusses “display 108 at which the raw and/or processed 2D images and 3D data can be presented”) before determining three-dimensional information based on an output from the light receiver (as taught by Spears, the 2D information is determined first in 908/910, while the 3D information is based on both the light receiver and the 2D images afterwards in 918).
It would have been obvious to one of ordinary skill in the art to combine Simek and Spears for the same reasons as discussed above.
Regarding claim 14, Simek and Spears disclose an imaging apparatus as discussed above, and Simek further discloses wherein the circuitry outputs the three-dimensional information to a destination that is different from the display (“3D modeling and navigation server device 112 can further be communicatively coupled to the 2D/3D panoramic capture device 102” as discussed in [0053], and “3D depth data… can be sent by the 2D/3D panoramic capture device 102 to… the 3D modeling and navigation server device 112” as discussed in [0056], while 112 is different from the display 108, see “the user device 106 and/or the 3D modeling and navigation server device 112” discussed in [0089]).
Regarding claim 15, Simek and Spears disclose an imaging apparatus as discussed above, and Simek further discloses wherein the circuitry is configured to determine three-dimensional information based on the output from the light receiver (“determine 3D data from one or more structured light sensor devices” discussed in [0101]).
Regarding claim 16, Simek and Spears disclose an imaging apparatus as discussed above, and Simek further discloses wherein the circuitry is further configured to output two-dimensional image information captured by the camera separately from the three-dimensional information (102 outputs the 2D information to 106, so it can be displayed on display 108, see [0058], while the “3D modeling and navigation server device 112 can further be communicatively coupled to the 2D/3D panoramic capture device 102” as discussed in [0053], and “3D depth data… can be sent by the 2D/3D panoramic capture device 102 to… the 3D modeling and navigation server device 112” as discussed in [0056]).
Regarding claim 17, Simek and Spears disclose an imaging apparatus as discussed above, and Simek further discloses wherein the circuitry is further configured to determine the three-dimensional information based on the output from the light receiver (“determine 3D data from one or more structured light sensor devices” discussed in [0101]) and the two-dimensional image information captured by the camera (“2D panoramic image of an environment can be combined with 3D panoramic depth data of the environment captured from the same location to determine depth information” discussed in [0046]).
Regarding claim 18, Simek and Spears disclose an imaging apparatus as discussed above, and Simek further discloses wherein the imaging optical elements include a fish-eye lens (as discussed above, 206 includes “fisheye lenses” as discussed in [0072]).
Regarding claim 21, Simek and Spears disclose an imaging apparatus as discussed above, and Spears further discloses wherein the camera includes a plurality of imaging optical elements (similar to “camera sensor 142 includes a wide-angle and/or fisheye lens” discussed in [0027]) arranged in different orientations from each other (as seen in Fig. 7, 742a is arranged on the left and 742n is arranged on the right),
the projector includes a plurality of projecting optical elements arranged in different orientations from each other (as seen in Fig. 7, 124a is arranged on the left and 124n is arranged on the right),
the light receiver includes a plurality of light receiving optical elements arranged in different orientations from each other (as seen in Fig. 7, 126a is arranged on the left and 126n is arranged on the right), and
the plurality of projecting optical elements and the plurality of light receiving optical elements are at different positions in a direction of the camera housing (seen in Fig. 7, see also the similar “camera sensor 142, emitter module 124, and receiver module 126 are configured on a same side of image capturing device 100. In one example, camera sensor 142, emitter module 124, and receiver module 126 may be aligned along a same baseline/axis” discussed in [0044]).
It would have been obvious to one of ordinary skill in the art to combine Simek and Spears for the same reasons as discussed above.
Regarding claim 22, Simek and Spears disclose an imaging apparatus as discussed above, and the combination further discloses wherein the circuitry is configured to communicate with an external device (eg. such as 112 of Simek, see also “communication component 402 can be configured to facilitate wired and/or wireless communication between the 2D/3D panoramic capture device 400 and an external device, such as the user device 106” as discussed in [0089]), without transmitting the two-dimensional image information that has been displayed on the display (Spears discloses outputting only 3D images without 2D images, see “3D image 210 may be provided to… at least one output device (e.g., display 145)” discussed in [0032]).
It would have been obvious to one of ordinary skill in the art to combine Simek and Spears for the same reasons as discussed above.
Claims 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Simek and Spears as applied to claim 18 above, and further in view of Kuroda et al. (US 2020/0096636).
Regarding claim 19, Simek and Spears disclose an imaging apparatus as discussed above, however fail to teach or suggest wherein the projecting optical elements include a wide-angle lens.
Kuroda (Fig. 1) discloses an imaging apparatus comprising:
a projector (22) to project light to the space (“22 emits light” discussed in [0091]), the projector including a projecting optical element (23);
a light receiver (25) to receive light reflected from the object (“irradiation light is radiated from the light-emitting diode 22 to the target object. Reflection light that is the irradiation light reflected on the target object is received by the TOF sensor 25” discussed in [0140]), the light receiver including a light receiving optical element (24); and
wherein the projecting optical element includes a wide-angle lens (23 is a “lens” and “adjusts distribution of light such that light radiated from the light-emitting diode 22 has a desired irradiation angle (e.g…. 100 degrees” as discussed in [0092], with “irradiation angle of the infrared light of the light-emitting diode 103-1 is set to 100 degrees” also discussed in [0197], and “wide viewing angle of 100 degrees” discussed in [0184]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Simek and Spears so the projecting optical elements each include a wide-angle lens as taught by Kuroda because this provides an increase the area of sensing (although trading off with increased noise, see [0184]).
Regarding claim 20, Simek, Spears, and Kuroda disclose an imaging apparatus as discussed above, and Kuroda wherein the plurality of light receiving optical elements include a wide-angle lens (“24 includes a wide-angle lens” discussed in [0093]) different from the wide-angle lens included in the projecting optical elements (as seen in Fig. 1, 23 and 24 are different lenses).
It would have been obvious to one of ordinary skill in the art to combine Simek, Spears, and Kuroda for the same reasons as discussed above.
Response to Arguments
Applicant's arguments filed 4-20-26 have been fully considered but they are not persuasive.
Regarding claim 1, the applicant argues that Simek fails to teach or suggest “the circuitry outputs the two-dimensional image information before outputting three-dimensional information” because Simek “requires that the 2D and 3D data be output at the same time.” The examiner respectfully disagrees. While Simek fails to explicitly teach the 2D information is output before the 3D information, Simek does not specify that the 2D and 3D data are necessarily output simultaneously. In fact, Simek teaches that the 2D and 3D data can be output “as it is generated” in close to “real time” (see [0058]), and further shows in Fig. 8 that the 2D and 3D data are generated in separate processes (eg. 902 and 804, respectively). As discussed above, Spears teaches that 2D data can be generated before 3D data (eg. 2D data in 908/910, and then 3D data afterwards in 918), and so the combination of Simek and Spears properly teaches the newly amended claim limitations (eg. Simek teaching the 2D data that is generated first in Spears in 908/910 is displayed first, and then afterwards displaying the 3D data as it is generated in 918 of Spears).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN M BLANCHA whose telephone number is (571)270-5890. The examiner can normally be reached Monday to Friday, 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chanh Nguyen can be reached at 5712727772. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONATHAN M BLANCHA/Primary Examiner, Art Unit 2623
Prosecution Insights