DETAILED ACTION
This Office Action for U.S. Patent Application 19/224,340 is responsive to communications filed on 3/2/26, in reply to the Non-Final Rejection of 10/1/25. Currently, claims 1-21 are pending.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 1/12/26 and 1/12/26 are in accordance with provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Response to Amendment
Applicant’s amendments to claim 1 and the addition of new claims 2-21 are acknowledged.
In view of Applicant’s amendments to claim 1, the nonstatutory double patenting rejection over U.S. Patent No. 11,677,920 is withdrawn. However, a new nonstatutory double patenting rejection appears below in response to the amendments to claim 1.
Response to Arguments
Applicant's arguments filed 3/2/26 have been fully considered but they are not persuasive.
Regarding claim 1, Applicant argues on pages 6-9 of the Response that Lynch and Rosenstein do teach “a horizontal rotatable mount configured to enable the at least one camera to move in a vertical y axis, the at least one camera being capable of capturing a plurality of images with mutually overlapping fields of view at different viewpoints”, as amended.
However, Rosenstein teaches that the head 160 may house a camera 320, which is situated on the neck 150. The neck 150 provides panning (i.e., movement in horizontal x axis) and tilting (i.e., movement in a vertical y axis) of the head 160 (and camera 320) with respect to the torso (Fig. 2; para[0109]-[0110]; para[0098]; Fig. 21C; Figs. 26A-26D; para [0172]). In addition, Rosenstein teaches in Fig. 2C an overlap between views captured by a color or stereo camera 320 (para[0187]; Fig. 21C).
Therefore, Lynch and Rosenstein teach all of the limitations of claim 1. In addition, please see the below-stated rejection of claim 1.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1-21 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. 12,348,698. Although the claims at issue are not identical, they are not patentably distinct from each other because the instant claims are very similar in scope to that of the patented claims. Table 1 below shows a comparison between the instant claims and the claims in the patent.
Table 1 – Comparison of Claims of App. No. 19/224,340 (Instant Application) and U.S. Patent No. 12,348,698 (Patented Claims)
App. No. 19/224,340
Claims (Emphasis Added for Differences)
U.S. Patent No. 12,348,698
Claims (Emphasis Added for Differences)
Claim 1
A device comprising:
a housing including:
at least one camera having a fisheye camera lens configured to capture 2D image data of an environment from a fixed location;
at least one depth sensor device including at least one light imaging detection and ranging (LiDAR) device;
a horizontal rotatable mount configured to enable the at least one camera to move in a vertical y axis, the at least one camera being capable of capturing a plurality of images with mutually overlapping fields of view at different viewpoints; and
at least one processor configured to map the 2D image data from the at least one camera and 3D depth data from the at least one depth sensor device to a common spatial 3D coordinate space based on known capture positions and orientations of the at least one camera and the at least one depth sensor device to facilitate associating 3D coordinates with respective visual features included in the 2D image data relative to the common spatial 3D coordinate space.
Claim 1
A device…comprising:
a housing including:
at least one camera having a fisheye camera lens configured to capture 2D image data of an environment from a fixed location…;
at least one depth sensor device including at least one light imaging detection and ranging (LiDAR) device,…; a horizontal rotatable mount configured to enable…the at least one camera to move about a vertical y axis…, the at least one camera being capable of capturing a plurality of images with mutually overlapping fields of view at different viewpoints; and
at least one processor configured to map the 2D image data…from the at least one camera and the 3D depth data from the at least one depth sensor device to a common spatial 3D coordinate space based on known capture positions and orientations of the at least one camera and the at least one depth sensor device.. to facilitate associating 3D coordinates with respective visual features included in the 2D image data relative to the common spatial 3D coordinate space;…
The remaining claims are analyzed similarly.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under pre-AIA 35 U.S.C. 103(a) are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-4, 7-9, 11-14, 17-19, and 21 are rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Lynch (U.S. Pub. No. 2013/0103303) in view of Rosenstein et al. (U.S. Pub. No. 2012/0185094; cited in the IDS filed 9/1/23).
In regard to claim 1, Lynch teaches a device comprising:
at least one depth sensor device including at least one light imaging detection and ranging (LiDAR) device (i.e., the system 150 receives a depthmap generated from an optical distancing system; the optical distancing system may be a LIDAR device, a stereoscopic camera, or a structured light device; optical distancing system 350) (Fig. 2B; para[0025], [0040]);
at least one processor (i.e., computer executable instructions may be logic encoded in one or more tangible media or one or more non-transitory tangible media for execution by the processors) (para[0075]) configured to map the 2D image data from the at least one camera (i.e., the image 200 is received in real time by a camera on the user device 100; S101 in Fig. 10) (Fig. 2A, 10; para[0038], [0064]-[0065]) and 3D depth data from the at least one depth sensor device (i.e., controller correlates the image data with a depthmap previously generated from the optical distancing system 350 and stored in memory 501 or memory 901; note: the depthmap being “captured” from the memory 501 based on correlation with the image 200) (Fig. 10; [0064]-[0066]) to a common spatial 3D coordinate space based on known capture positions and orientations of the at least one camera and the at least one depth sensor device (i.e., Fig. 2A illustrates a predetermined camera angle that the image 200 (e.g. “the 2D image data”) was collected by a camera 250; the camera also records its specific position location (e.g. geo-coordinates); the predetermined camera angle may be used to align the depth data to the panoramic image (e.g., “map…the 2D image data…and 3D depth data…to common spatial 3D coordinate space”); the image 200 may be a planar image (e.g., 2D image in a plane); the optical distancing system 350 may be coupled with an inertial measurement unit (IMU) to associate the optical distance data (e.g., “the 3D depth data”) with the geo-located position of the optical distancing system 350) (Figs. 2A, 2B; para[0039]-[0040]) to facilitate associating 3D coordinates with respective visual features included in the 2D image data relative to the common spatial 3D coordinate space (i.e., the C-C cross section plane or top down view 600 in Fig. 4A shows the depth values are defined to correspond to the sign 207, house 201, and tree 203; the C-C cross section plane or top down view 600 in Fig. 4B shows the depth values are defined to correspond to the sign 207 and the wall 205) (Figs. 4A, 4B; para[0044]).
However, Lynch does not explicitly teach at least one camera configured to capture 2D image data of an environment from a fixed location;
and does not explicitly teach
a horizontal rotatable mount configured to enable the fisheye camera lens of the at least one camera to move in a vertical y axis, the at least one camera being capable of capturing a plurality of images with mutually overlapping fields of view at different viewpoints.
In the same field of endeavor, Rosenstein teaches at least one configured to capture 2D image data of an environment from a fixed location (i.e. the field of view 452 of the imaging sensor 450…can be enlarged to 360 degrees by optics, such as …fisheye…lenses) (Fig. 17A, 17B; para[0172]);
and teaches
a horizontal rotatable mount configured to enable the fisheye camera lens of the at least one camera to move in a vertical y axis (i.e. Figs. 1-3 and 7, the torso 140 supports the neck 150, which provides panning (i.e., movement in horizontal x axis) and tilting (i.e., movement in a vertical y axis) of the head 160 with respect to the torso; the neck 150 includes a rotator and a tilter 154; the rotator 152 includes electrical connectors or contacts that allow continuous 360° rotation of the head 160 with respect to the torso 140; the head 160 may house…a camera 320; the drive system 200 provides omni-directional and/or holonomic motion control of the robot 100; the ability to move in substantially any planar direction, i.e. side-to-side (lateral), forward/back, and rotational; the field of view 452 of the imaging sensor 450…can be enlarges to 360 degrees by optics, such as …fisheye…lenses;) (Fig. 2; para[0109]-[0110]; para[0098]; Fig. 21C; Figs. 26A-26D; para [0172]), the at least one camera being capable of capturing a plurality of images with mutually overlapping fields of view at different viewpoints (i.e. a color or stereo camera 320; overlap between views) (para[0187]; Fig. 21C).
It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the invention, to combine the teachings of Lynch and Rosenstein because Rosenstein teaches a mobile imaging device to capture three-dimensional depth images to determine the locations of objects in a scene (See, for example, para[0004]-[0006] of Rosenstein). Specifically, Rosenstein teaches using a variety of image sensors (e.g., omni-directional, fisheye, catadioptric, panamorph mirrors and lenses) in order to enlarge the field of view 452 (up to 360 degrees) (See, for example, para[0172] of Rosenstein). It would have been obvious to one of ordinary skill in the art that capturing a larger field of view in an image such as taught in Rosenstein would allow for a more efficient capture of the environment. Therefore, it would have been obvious to combine the teachings of Lynch and Rosenstein.
In regard to claim 2, Lynch and Rosenstein teach all of the limitations of claim 1 as discussed above. However, Lynch does not explicitly teach further including a vertical rotatable mount configured to enable the at least one camera to rotate about a horizontal x axis.
In the same field of endeavor, Rosenstein teaches further including a vertical rotatable mount configured to enable the at least one camera to rotate about a horizontal x axis (i.e. Figs. 1-3 and 7, the torso 140 supports the neck 150, which provides panning and tilting of the head 160 with respect to the torso; the neck 150 includes a rotator and a tilter 154; the tilter 154 may move the head…with respect to the torso 140; the head 160 may house…a camera 320) (para[0109]-[0110]).
It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the invention, to combine the teachings of Lynch and Rosenstein for the same reasons as those discussed above for claim 1.
In regard to claim 3, Lynch and Rosenstein teach all of the limitations of claims 1 and 2 as discussed above. However, Lynch does not explicitly teach wherein the horizontal rotatable mount and the vertical rotatable mount each include at least one motor configured to rotate the at least one camera.
In the same field of endeavor, Rosenstein teaches wherein the horizontal rotatable mount and the vertical rotatable mount each include at least one motor configured to rotate the at least one camera (i.e. Figs. 1-3 and 7, the torso 140 supports the neck 150, which provides panning and tilting of the head 160 with respect to the torso; the neck 150 includes a rotator and a tilter 154; the rotator 152 includes electrical connectors or contacts that allow continuous 360° rotation of the head 160 with respect to the torso 140; the head 160 may house…a camera 320) (para[0109]-[0110]).
It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the invention, to combine the teachings of Lynch and Rosenstein for the same reasons as those discussed above for claim 1.
In regard to claim 4, Lynch and Rosenstein teach all of the limitations of claim 1 as discussed above. However, Lynch does not explicitly teach wherein the at least one camera includes lens that provides a field of view from about 100 degrees to about 195 degrees.
In the same field of endeavor, Rosenstein teaches wherein the at least one camera includes lens that provides a field of view from about 100 degrees to about 195 degrees(i.e. the field of view 452 of the imaging sensor 450 having a viewing angle θv less than 360 can be enlarged to 360 degrees by optics, such as …fisheye…lenses) (para[0172]).
It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the invention, to combine the teachings of Lynch and Rosenstein for the same reasons as those discussed above for claim 1.
In regard to claim 7, Lynch and Rosenstein teach all of the limitations of claim 1 as discussed above. However, Lynch does not explicitly teach wherein the at least one camera is a color video camera capable of capturing color video of the environment.
In the same field of endeavor, Rosenstein teaches wherein the at least one camera is a color video camera capable of capturing color video of the environment (i.e., the controller 500 may receive image data from an image sensor 450, such as a color, black and white, or IR camera; the image sensor 450 of the robot 100 can capture images or live video) (para[0182], [0266]).
It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the invention, to combine the teachings of Lynch and Rosenstein for the same reasons as those discussed above for claim 1.
In regard to claim 8, Lynch and Rosenstein teach all of the limitations of claim 1 as discussed above. In addition, Lynch teaches wherein the at least one processor is further configured to determine each capture location relative to each other using positional tracking (i.e., the optical distancing system 350 may be coupled with an inertial measurement unit (IMU) and/or an inertial navigation system (INS) in order to provide a geographic reference to the optical distance data; the optical distancing system 350 may be coupled with an inertial measurement unit (IMU) to associated the optical distance data with the geo-located position of the optical distancing system 350) (Fig. 2B; para[0040]).
In regard to claim 9, Lynch and Rosenstein teach all of the limitations of claims 1 and 8 as discussed above. In addition, Lynch teaches wherein the positional tracking includes inertial position tracking (i.e., the optical distancing system 350 may be coupled with an inertial measurement unit (IMU) to associated the optical distance data with the geo-located position of the optical distancing system 350) (Fig. 2B; para[0040]).
In regard to claims 11-14 and 17-19, the claims recite analogous limitations to claims 1-4 and 7-9 above, and are therefore rejected on the same premise.
In regard to claim 21, the claim recites analogous limitations to claim 1 above, and is therefore rejected on the same premise.
Claims 5 and 15 are rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Lynch (U.S. Pub. No. 2013/0103303) in view of Rosenstein et al. (U.S. Pub. No. 2012/0185094; cited in the IDS filed 9/1/23), further in view of Baker et al. (U.S. Pub. No. 2012/0105574; cited in the IDS filed 9/1/23).
In regard to claim 5, Lynch and Rosenstein teach all of the limitations of claim 1 as discussed above. However, Lynch and Rosenstein do not explicitly teach wherein the at least one processor is further configured to merge and align a plurality of 2D images captured by the at least one camera to generate a panoramic 2D image of the environment.
In the same field of endeavor, Baker teaches wherein the at least one processor is further configured to merge and align a plurality of 2D images captured by the at least one camera to generate a panoramic 2D image of the environment (i.e. pair of panoramic images (200, 205) that are combined to provide stereoscopic perspective; the first panoramic image (200) was captured by the first array (100) and the second panoramic image (205) was captured by the second array) (Figs. 2A and 2B; para[0054]-[0056]).
It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the invention, to combine the teachings of Lynch and Rosenstein with those of Baker because Baker teaches creating panoramic images out of, for example, 10 individually taken images and stitching them together in order to later provide stereoscopic images (See, for example, Figs. 2A, 2B; para[0054]-[0056] of Baker). Therefore, it would have been obvious to combine Lynch and Rosenstein with Baker.
In regard to claim 15, the claim recites analogous limitations to claim 5 above, and is therefore rejected on the same premise.
Claims 6 and 16 are rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Lynch (U.S. Pub. No. 2013/0103303) in view of Rosenstein et al. (U.S. Pub. No. 2012/0185094; cited in the IDS filed 9/1/23), further in view of Baker et al. (U.S. Pub. No. 2012/0105574; cited in the IDS filed 9/1/23) and Wolberg et al. (U.S. Pub. No. 2008/0310757; cited in the IDS filed 9/1/23).
In regard to claim 6, Lynch, Rosenstein, and Baker teach all of the limitations of claim 1 and 5 as discussed above. However, Lynch, Rosenstein, and Baker do not explicitly teach wherein the at least one processor is further configured to merge and align the 3D depth data based on information from the at least one depth sensor device to generate a 3D image of the environment.
In the same field of endeavor, Wolberg teaches wherein the at least one processor is further configured to merge and align the 3D depth data based on information from the at least one depth sensor device to generate a 3D image of the environment (i.e. multiview geometry for texture mapping 2D images onto 3D range data; the invention is discussed in more detail in para[0047]-[0074]; Fig. 2; para[0063] discusses further alignment of 3D models) (para[0036]-[0041]).
It would have been obvious to a person having ordinary skill in the art, at the time of applicant's invention, to combine the teachings of Lynch, Rosenstein, and Baker with those of Wolberg because Wolberg teaches a 3D sensor configured to generate a plurality of 3D range scans of a scene and a 2D sensor configured to generate a plurality of 2D images, where the 3D range scans and the 2D photographs are respectively used to generate a pair of 3D models of the scene (See, for example, Fig. 2, para[0014], [0047]). Therefore, it would have been obvious to combine the teachings of Lynch, Rosenstein, and Baker with those of Wolberg.
In regard to claim 16, the claim recites analogous limitations to claim 6 above, and is therefore rejected on the same premise.
therefore rejected on the same premise.
Claims 10 and 20 are rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Lynch (U.S. Pub. No. 2013/0103303) in view of Rosenstein et al. (U.S. Pub. No. 2012/0185094; cited in the IDS filed 9/1/23), further in view Wolberg et al. (U.S. Pub. No. 2008/0310757; cited in the IDS filed 9/1/23).
In regard to claim 10, Lynch and Rosenstein teach all of the limitations of claims 1 and 8 as discussed above. However, Lynch and Rosenstein do not explicitly teach wherein the positional tracking includes utilizing simultaneous localization and mapping (SLAM) analysis.
In the same field of endeavor, Wolberg teaches wherein the positional tracking includes utilizing simultaneous localization and mapping (SLAM) analysis (i.e. second model consists of a sparse 3D point cloud, produced by applying a Multiview geometry (structure-from-motion) algorithm, which is also known as SLAM, or Simultaneous Localization and Mapping, directly on a sequence of 2D photographs to simultaneously recover the camera motion and the 3D positions of image features) (para[0036]).
It would have been obvious to a person having ordinary skill in the art, at the time of applicant's invention, to combine the teachings of Lynch and Rosenstein with those of Wolberg because Wolberg teaches a 3D sensor configured to generate a plurality of 3D range scans of a scene and a 2D sensor configured to generate a plurality of 2D images, where the 3D range scans and the 2D photographs are respectively used to generate a pair of 3D models of the scene (See, for example, Fig. 2, para[0014], [0047]). Therefore, it would have been obvious to combine the teachings of Lynch and Rosenstein with those of Wolberg.
In regard to claim 20, the claim recites analogous limitations to claim 10 above, and is therefore rejected on the same premise.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Kristin Dobbs whose telephone number is (571)270-7936. The examiner can normally be reached Monday and Thursday 9:30am-5:30pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sathyanarayanan Perungavoor can be reached at (571)272-7455. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
KRISTIN DOBBS
Examiner
Art Unit 2488
/KRISTIN DOBBS/Examiner, Art Unit 2488
/SATH V PERUNGAVOOR/Supervisory Patent Examiner, Art Unit 2488
Prosecution Insights