Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continuation application claiming priority to provisional dated 10/15/2019.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1, 3-10, and 12-19 are rejected under 35 U.S.C. 103 as being unpatentable over IVANOV; Pavel et al. US 20170371024 A1 in view of US 20190287311 A1 BHATNAGAR; Tushar Cyril et al.
Regarding Claims 1, 10 and 19, The combination teaches A portable device (See Ivanov [0032]-[0033] Fig. 1 Apparatus 100 Fig. 3 Collecting Devices Fig. 4 Mobile Device) configured to operate within a three-dimensional (3D) environment (See Ivanov [0095] three-dimensional environment multi-level building) and display virtual content of a cross reality system (See Ivanov [0054] [0097] map display),
the 3D environment comprising a plurality of locations associated with a plurality of features extracted from image data and a wireless fingerprint linked with a location of the plurality of locations (See Ivanov [0032]-[0033] Fig. 5 504, 512, 513, [0103] “each mobile device 411, 412 presents a map, in which the listed locations of access points and the associated quality of the radio models are indicated, on a display of mobile device 411, 412 to the user of mobile device 411, 412. (action 513)”);
the wireless fingerprint comprising first network access point information obtained from one or more network access points at a first time (Fig. 5 steps 501-503 and 511 initial collection and aggregation of fingerprint data ¶88--¶98, see e.g. [0091] “..a collecting agent inputs the locations for the fingerprints manually, e.g. using indoor map of the building..”; see also [0095] “..The aggregation may further comprise a mapping of the results of measurements in the fingerprints to grid points of one or more grids..”; See e.g. [0046]-[0047] radio models with locations and grid points),
the portable device comprising:
at least one processor (See Ivanov [0032]-[0033] Fig. 1 Processor 101 Fig. 7, [0148] Processor 702);
wireless hardware (¶91 “A WLAN component of mobile device 411, 412 detects radio signals from WLAN access points 440 in the environment and performs radio measurements on these signals at the same regular intervals.”);
and computer executable instructions configured to, when executed by the at least one processor, perform a method (See Ivanov Figs. 2-6) comprising:
obtaining second network access point information from the one or more network access points at a second time after the first time using the wireless hardware (See Ivanov Fig. 5 steps 501-508 at server along with 511-517 at device and ¶130 “Next time when a new survey of the localization site starts, a new instance of the synchronization process may be initialized with previously collected, generated and determined data, and information about coverage, sufficiency and validity of the data that has been collected so far can be sent to all mobile devices 411, 412 that are used for the new survey in the beginning of the collection process, that is, in action 501.”);
[repeatedly] (See Ivanov Fig. 5, [0091] regular intervals) identifying the location from among the plurality of locations of the 3D environment (See Ivanov Fig. 5, [0091], [0092] teaching identifying location of collective device a location at a particular time and measurement by inputting by agent or by GNSS via cellular assistance)
and based on selecting the location, updating the wireless fingerprint linked with the location of the 3D environment by merging the second network access point information obtained at the second time with the first network access point information obtained at the first time (See Ivanov Fig. 5 all steps culminating in ¶129 “[0129] Finally, server 400 supplements the previously stored updated data with newly determined data and the data is stored as positioning assistance data, e.g. in memory 403 or in an external memory. (action 508) The data that is eventually stored may be in various forms. . . . “ ).
Ivanov does not explicitly disclose the portable device comprising an image sensor;
extracting one or more features from image data obtained by the image sensor;
identifying the location from among the plurality of locations of the 3D environment based on correspondence between the one or more features extracted from the image data obtained by the image sensor and features of the plurality of features of the 3D environment;
Bhatnager teaches the portable device (Bhatnager Fig. 1 device 10) comprising an image sensor (Bhatnager Fig. 1, Fig. 2 depth camera 21, RGB cameras 18, image data 121A 118A);
extracting a plurality of features from image data obtained by the image sensor (Bhatnager Fig. 2 Features 111A, Feature library 11);
identifying the location from among the plurality of locations of the 3D environment based on correspondence between the one or more features extracted from the image data obtained by the image sensor and features of the plurality of features of the 3D environment (Bhatnager Fig. 4A and Fig. 5 Map Data Request 48; [0033] “The server computing device 200 may be configured with a wireless transceiver 230 that wirelessly communicates with the display device 30 to receive fingerprint requests 52 and map data requests 48 from the display device 30 and transmits fingerprint data 54 and matching map data sets 66 to the display device 30” [0042] the client fingerprint program 38 may make a determination that a plurality of candidate fingerprints are the closest match to the current fingerprint, then generate and send a map data request 48 for the map data set or plurality of map data sets within the global map data set 158 corresponding to the identified keyframes linked to the closest matching candidate fingerprints” and also [0025] Data captured by the visible light cameras 18, the depth camera 21, and the inertial motion unit 19 can be used to perform simultaneous location and mapping (SLAM) within the physical environment 9, to thereby produce a map of the physical environment including a mesh of reconstructed surfaces, and to locate the computing device 10 within the map of the physical environment 9. The location of the computing device 10 is computed in six degrees of freedom, which is important to displaying world-locked holograms 50 on the at least partially see through display 20.);
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art, to modify the invention of Ivanov to include the noted teachings of Bhatanager, in order to quickly orient itself or determine its own position relative to a three-dimensional coordinate space of the physical environment by efficiently loading the appropriate map data sets to accelerate the coarse relocalization step (Bhatanager [0003]).
Further regarding Claim 19, the combination teaches creating a tracking map of the 3D environment by extracting a plurality of features from image data obtained by the image sensor, the tracking map comprising a plurality of locations (Bhatanager ¶27-28 “..As the computing device 10 moves throughout the physical environment 9, it maps the environment and stores its aggregated knowledge of the environment as map data 86. . .”; Fig. 5 Global map data set 158 key frames 160 pose graphs 162 [0030]-[0032] feature mapping algorithm), each location of the plurality of locations linked with at least one feature of the plurality of features extracted from the image data obtained by the image sensor (Bhatanager ¶27-28 “one possible SLAM algorithm for simultaneous location and mapping by computing device 10. According to the SLAM algorithm, a predictive corrective model 28 is typically applied, which may, for example, be a Kalman filter. In each frame (i.e., timestep), a predicted pose 29 in a next frame is computed based on odeometry data 119A from IMU 19 by an integration engine 27, and then a correction to the predicted pose 29 is computed by the predictive corrective model (e.g., Kalman filter) 28 based on estimated and observed positions of features in the physical environment 9 sensed by sensors such as cameras 18 and depth camera 21. . . or each detected feature, the location (e.g., coordinate area) and type of the feature are stored as observation data 17 associated with each frame. It will be appreciated that dozens or hundreds of such features may be recognized in an image, and the collection of these observations 17 of features may be referred to informally as a pointcloud of detected features in the image..”);
Regarding Claims 3 and 12, The combination teaches wherein obtaining network access point information comprises triggering a scan for network access points (See Ivanov [0091] “A WLAN component of mobile device 411, 412 detects radio signals from WLAN access points 440 in the environment”), and further regarding Claim 11, The combination teaches wherein the scan for network access points is triggered when an amount of time passed since a previous scan of network access points was performed exceeds a threshold (See Ivanov [0091] “A WLAN component of mobile device 411, … performs radio measurements on these signals at the same regular intervals”).
Regarding Claims 4 , The combination teaches wherein obtaining network access point information comprises receiving the network access point information pushed from wireless hardware components after a scan for network access points (See Ivanov [0091] “A WLAN component of mobile device 411, 412 detects radio signals from WLAN access points 440 in the environment and performs radio measurements on these signals at the same regular intervals”).
Regarding Claims 5 and 13, The combination teaches wherein obtaining second network access point information comprises obtaining access point identifiers of the network access points (See Ivanov [0064], [0091] BSSID).
Regarding Claims 6 and 14, The combination teaches wherein obtaining second network access point information further comprises obtaining one or more signal strength indicator values for access points identified by the access point identifiers (See Ivanov [0064], [0091] [0091] “results of the measurements contain an identifier (ID) of each observed access points, like a BSSID, and a received signal strength (RSS) value”).
Regarding Claim 7, The combination teaches wherein the access point identifiers are Basic Service Set Identifiers (BSSIDs) (See Ivanov [0064], [0091] BSSID).
Regarding Claim 8, The combination teaches wherein obtaining second network access point information further comprises filtering, clustering, and/or normalizing the network access point information (See Ivanov [0095] “..The grid point to which the measurement results of a particular fingerprint are mapped could be the grid point that corresponds to a real location that is closest to the horizontal location indicated in the fingerprint..” describing a clustering technique and [0112] “..Each mobile device 411, 412 filters the list to obtain an indication of those suspicious fingerprints..”).
Consider Claim 9
The combination teaches The portable device of claim 1, wherein:
each location of the plurality of locations of the 3D environment comprises information configured to orient the portable device relative to the 3D environment (Ivanov ¶97 “Each of the mobile devices 411, 412 presents a map, in which the listed locations are highlighted, on a display of the mobile device 411, 412 to the user of the mobile device 411, 412. (action 512) The presented map may include for example the outlines of a floor, possibly with an indication of inner walls, open spaces—as in the case of stair cases or elevators —, etc. The highlighted locations indicate to the users at which locations of the localization site fingerprints have already been collected and from which locations of the localization site fingerprints are still missing. Thus, they constitute a guidance to the user for the collection of additional finger” and See also Bhatnager [0030] pose is typically a unit vector with an origin at a predetermined location (x, y, and z) and extending in a predetermined orientation (pitch, yaw, and roll) in the physical space, and is calculated as described in relation to FIG. 2. The origin corresponds to a predetermined fixed location on the computing device, such as equidistant between the left and right displays, and the unit vector extends outward in at a fixed attitude from the display, in a gaze direction of the user.).
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art, to modify the combination include the noted teachings of Bhatanager, in order to o quickly orient itself or determine its own position relative to a three-dimensional coordinate space of the physical environment by efficiently loading the appropriate map data sets to accelerate the coarse relocalization step (Bhatanager [0003]).
Regarding Claim 15, The combination teaches updating the information stored in association with the selected location within the map comprises storing the one or more signal strength indicator values in association with the access point identifiers (See Ivanov [0064], [0091] [0091] “results of the measurements contain an identifier (ID) of each observed access points, like a BSSID, and a received signal strength (RSS) value” [0093]-[0094]” Sever 400 obtains fingerprints from a plurality of mobile devices 411, 412. (action 502)”).
Regarding Claim 16, The combination teaches wherein updating the wireless fingerprint linked with the location comprises identifying a subset of the access point identifiers to be excluded (See Ivanov [0119]-[0122] suspicious APs identified).
Regarding Claim 17, The combination teaches wherein the subset of access point identifiers to be excluded is based at least on the one or more signal strength indicator values (See Ivanov [0119]-[0122] suspicious APs locations and change determined based on signal strengths).
Regarding Claim 18, The combination teaches wherein storing the one or more signal strength indicator values comprises storing an average of multiple signal strength indicator values in association with the access point identifiers (Ivanov [0095] “If the measurement results for the same WLAN access point from several fingerprints would be mapped to the same grid point, an average value of the received signal strength (RSS) could be used, for example.”).
Double Patenting
The non-statutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 11632679 (Parent).
Although the claims at issue are not identical, they are not patentably distinct from each other because claims 1-20 of issued patent are similar and any elements that are different require only ordinary skill or a broader level of generalization of the disclosed claims. Table below compares the independent claim 1 to exemplify the remaining claims as well.
Instant Application Claim 1
17/071,879 --- US 11632679 B2 (Parent)
1. A portable device configured to operate within a three-dimensional (3D) environment and display virtual content of a cross reality system, the 3D environment comprising a plurality of locations associated with a plurality of features extracted from image data and a wireless fingerprint linked with a location of the plurality of locations, the wireless fingerprint comprising first network access point information obtained from one or more network access points at a first time, the portable device comprising:
at least one processor;
an image sensor;
wireless hardware; and
computer executable instructions configured to, when executed by the at least one processor, perform a method comprising:
obtaining second network access point information from the one or more network access points at a second time after the first time using the wireless hardware;
extracting one or more features from image data obtained by the image sensor;
identifying the location from among the plurality of locations of the 3D environment based on correspondence between the one or more features extracted from the image data obtained by the image sensor and features of the plurality of features of the 3D environment; and
based on selecting the location, updating the wireless fingerprint linked with the location of the 3D environment by merging the second network access point information obtained at the second time with the first network access point information obtained at the first time.
1. A portable device configured to operate within a three-dimensional (3D) environment and display virtual content of a cross reality system, the portable device comprising:
at least one processor;
an image sensor; and
computer executable instructions configured to, when executed by the at least one processor, perform a method comprising:
forming a map of the 3D environment based on image data produced by the image sensor as the portable device moves in the 3D environment, the map of the 3D environment comprising a plurality of feature descriptors and having a first coordinate frame;
maintaining a wireless fingerprint associated with the portable device by repeatedly:
obtaining network access point information from network access points sending wireless signals received by the portable device at a location within the 3D environment; selecting a location within the map corresponding to the location of the portable device within the 3D environment; and
updating information stored in association with the selected location within the map, based on the obtained network access point information; sending the wireless fingerprint and the plurality of feature descriptors to a localization service; and
receiving, from the localization service, a coordinate transformation between the first coordinate frame of the map of the 3D environment and a coordinate frame of a stored map, wherein the localization service generates the coordinate transformation based at least in part on the wireless fingerprint and the plurality of feature descriptors sent to the localization service.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-20 of U.S. Patent No. 12170910 (Parent).
Although the claims at issue are not identical, they are not patentably distinct from each other because claims 1-20 of issued patent are similar and any elements that are different require only ordinary skill or a broader level of generalization of the disclosed claims. Table below compares the independent claim 1 and claim 19 to exemplify the remaining claims as well.
Instant Application
18/179,275 --- US 12170910 (parent)
Claim 1
1. A portable device configured to operate within a three-dimensional (3D) environment and display virtual content of a cross reality system, the 3D environment comprising a plurality of locations associated with a plurality of features extracted from image data and a wireless fingerprint linked with a location of the plurality of locations, the wireless fingerprint comprising first network access point information obtained from one or more network access points at a first time, the portable device comprising:
at least one processor;
an image sensor;
wireless hardware; and
computer executable instructions configured to, when executed by the at least one processor, perform a method comprising:
obtaining second network access point information from the one or more network access points at a second time after the first time using the wireless hardware;
extracting one or more features from image data obtained by the image sensor;
identifying the location from among the plurality of locations of the 3D environment based on correspondence between the one or more features extracted from the image data obtained by the image sensor and features of the plurality of features of the 3D environment; and
based on selecting the location, updating the wireless fingerprint linked with the location of the 3D environment by merging the second network access point information obtained at the second time with the first network access point information obtained at the first time.
Claim 19:
19. A portable device configured to operate within a three-dimensional (3D) environment and display virtual content of a cross reality system, the 3D environment comprising a plurality of features, the portable device comprising:
at least one processor;
an image sensor;
wireless hardware; and
computer executable instructions configured to, when executed by the at least one processor, perform a method comprising:
creating a tracking map of the 3D environment by extracting a plurality of features from image data obtained by the image sensor, the tracking map comprising a plurality of locations, each location of the plurality of locations linked with at least one feature of the plurality of features extracted from the image data obtained by the image sensor;
repeatedly identifying a location from the plurality of locations of the tracking map using the plurality of features extracted from the image data obtained by the image sensor and updating a wireless fingerprint linked with the location using first network access point information obtained with the wireless hardware at a first time, the wireless fingerprint comprising second network access point information obtained at a second time prior to the first time.
1. A portable device configured to operate within a three-dimensional (3D) environment and display virtual content of a cross reality system, the portable device comprising:
at least one processor;
an image sensor configured to output image data; and
computer executable instructions configured to, when executed by the at least one processor, perform a method comprising:
extracting a plurality of features from the image data;
forming a map of the 3D environment based on the image data produced by the image sensor as the portable device moves in the 3D environment, the map of the 3D environment comprising a first coordinate frame and a plurality of persistent poses associated with feature descriptors of features of the plurality of features extracted from the image data and associated with locations within the map of the 3D environment;
maintaining a wireless fingerprint associated with the map of the 3D environment by repeatedly:
obtaining network access point information from network access points sending wireless signals received by the portable device at a location within the 3D environment;
selecting a persistent pose of the plurality of persistent poses at a location within the map of the 3D environment based on correspondence between features of the plurality of features and features associated with the persistent pose; and
based on selecting the persistent pose, updating a wireless fingerprint stored in association with the location within the map of the 3D environment by merging network access point information obtained at a second time with network access point information obtained at a first time and linked with the selected pose to produce a merged wireless fingerprint linked with the selected persistent pose;
for a set of features of the plurality of features associated with a position of the portable device, determining the location within the map associated with the features and sending to a localization service the merged wireless fingerprint stored in association with the determined location within the map and a plurality of feature descriptors associated with features of the set of features; and
based on sending the merged wireless fingerprint associated with the determined location and the plurality of feature descriptors to the localization service, receiving, from the localization service, a coordinate transformation between the first coordinate frame of the map of the 3D environment and a coordinate frame of a stored map.
Allowable Subject Matter
Claims 2, 11 and 20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Claims recite the following subject matter which is not found in the cited prior art in combination with the other features of the claims:
for the one or more features of the plurality of features extracted from the image data obtained by the image sensor, sending to a localization service at least one feature of the one or more features extracted from the image data obtained by the image sensor and the updated wireless fingerprint linked with the location of the 3D environment ; and
based on sending to the localization service the at least one feature and the updated wireless fingerprint linked with the location, receiving from the localization service a coordinate transformation between a first coordinate frame of the 3D environment and a coordinate frame of a stored map.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to UMAIR AHSAN whose telephone number is (571)272-1323. The examiner can normally be reached Monday - Friday 10-5 PM EST or by emailing UMAIR.AHSAN@USPTO.GOV.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alison Slater can be reached on (571) 270-0375. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/UMAIR AHSAN/Primary Examiner, Art Unit 2647
Prosecution Insights