Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “input module”, “matching modulate” and “output module” of claim 18 and “an input means” of claim 21
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof. A processor 202 comprising “an input module” 204; “matching module” 210; “output module” 212 and “input means” 216 See paragraphs 0072 and 0073.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “214” has been used to designate both ( and (Visual camera). Character 216 has been used to designate (Memory), (Display screen) and (input means)
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 1,2 and 11 are objected to because of the following informalities: in claim 1, “overlay information with a displayed representation of the environment” indicates some displaying/visualization. However, no means to perform the displaying is defined.. (see for example claim 16)Furthermore, the Term “overlay information” is not clearly defined. The same applies to claim 2 and 11. Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4,6-19,21-22 are rejected under 35 U.S.C. 102(a)(1) being anticipated by Lee (US2017/0019658 A1).
Regarding claim 1, Lee teaches: A computer-implemented method of localization of a portable imaging apparatus in an environment, the method comprising:
receiving imaging data indicative of the environment, the imaging data captured by the portable imaging apparatus located at a position within the environment; (Lee see fig. 1 and para. 0016] The following description is intended to convey a thorough understanding of the present disclosure by providing a number of specific embodiments and details involving the determination of a relative position or relative orientation of an electronic device based on image-based identification of objects in a local environment of the electronic device)
identifying a location of the portable imaging apparatus in the environment by matching an identified object feature in the imaging data with a corresponding object feature in a prior map of the environment, the prior map linked to a three-dimensional model of the environment; (0029] The relative position/orientation information obtained by the electronic device 100 from the image data captured by the imaging cameras 114, 116, and 118 can be used to support any of a variety of location-based functionality. The relative position/orientation information can be used by the electronic device 100 to support visual odometry or other SLAM functionality. As an example, the electronic device 100 can map the local environment 112 and then use this mapping to facilitate the user's navigation through the local environment 112, such as by displaying to the user a floor plan generated from the mapping information and an indicator of the user's current location relative to the floor plan as determined from the current relative position of the electronic device 100. ) and
outputting overlay information with a displayed representation of the environment viewed from the identified location of the portable imaging apparatus, the overlay information retrieved from the prior map and indicative of prior stored data relating to a three-dimensional model feature in the three-dimensional model.( [0030] Moreover, the relative position/orientation information obtained by the electronic device 100 can be combined with supplemental information 144 to present an augmented reality (AR) view of the local environment 112 to the user 110 via the display 108 of the electronic device 100. This supplemental information 144 can include one or more AR databases locally stored at the electronic device 100 or remotely accessible by the electronic device 100 via a wired or wireless network. And para 0031.. Accordingly, the electronic device 100 can capture video imagery of a view of the local environment 112 via the imaging camera 116, determine a relative orientation/position of the electronic device 100 as described above and herein, and determine the position and orientation of electrical wiring located within the walls present in the view of the local environment. The electronic device 100 then can generate a graphical overlay with visual representations of the electrical wiring positioned and oriented relative to corresponding spatial features (e.g., the corners 124, 126, and 128) identified in the video imagery. As illustrated in FIG. 1, the graphical overlay can include colored dashed lines 152 and 154 representing electrical wiring in the current view and description balloons 156 and 158 to provide descriptions of the electrical wiring, such as wiring type, an identifier associated with the wiring, and the building components powered by the corresponding wiring. The electronic device 100 then jointly presents the graphical overlay and the video imagery at the display 108 so as to present the user 110 with a graphical representation 160 of the location of electrical wiring within the current view of the local environment 112 as captured by the narrow angle imaging camera 116. As the electronic device 100 moves relative to the previous view, the electronic device 100 updates the graphical overlay so as to reflect the changed perspective. Moreover, the head tracking data 140 can be used to detect changes in the position of the head 122 of the user 110 relative to the display 108, in response to which the electronic device 100 can adjust the displayed graphical representation 160 so as to reflect the changed viewing angle of the user 110 relative to the display 108..)
Regarding claim 2, The computer-implemented method of claim 1,
wherein the method further comprises: identifying a view direction of the portable imaging apparatus in the environment by matching the identified object feature in the imaging data with the corresponding object feature in [[a]] the prior map of the environment, and wherein outputting the overlay information with the displayed representation of the environment comprises outputting the overlay information with the displayed representation of the environment viewed from the identified location of the portable imaging apparatus in the identified view direction.
( Lee teaches [0030] Moreover, the relative position/orientation information obtained by the electronic device 100 can be combined with supplemental information 144 to present an augmented reality (AR) view of the local environment 112 to the user 110 via the display 108 of the electronic device 100. This supplemental information 144 can include one or more AR databases locally stored at the electronic device 100 or remotely accessible by the electronic device 100 via a wired or wireless network. And para 0031.. Accordingly, the electronic device 100 can capture video imagery of a view of the local environment 112 via the imaging camera 116, determine a relative orientation/position of the electronic device 100 as described above and herein, and determine the position and orientation of electrical wiring located within the walls present in the view of the local environment. The electronic device 100 then can generate a graphical overlay with visual representations of the electrical wiring positioned and oriented relative to corresponding spatial features (e.g., the corners 124, 126, and 128) identified in the video imagery. As illustrated in FIG. 1, the graphical overlay can include colored dashed lines 152 and 154 representing electrical wiring in the current view and description balloons 156 and 158 to provide descriptions of the electrical wiring, such as wiring type, an identifier associated with the wiring, and the building components powered by the corresponding wiring. The electronic device 100 then jointly presents the graphical overlay and the video imagery at the display 108 so as to present the user 110 with a graphical representation 160 of the location of electrical wiring within the current view of the local environment 112 as captured by the narrow angle imaging camera 116. As the electronic device 100 moves relative to the previous view, the electronic device 100 updates the graphical overlay so as to reflect the changed perspective. Moreover, the head tracking data 140 can be used to detect changes in the position of the head 122 of the user 110 relative to the display 108, in response to which the electronic device 100 can adjust the displayed graphical representation 160 so as to reflect the changed viewing angle of the user 110 relative to the display 108..)
Regarding claim 3, The computer-implemented method of claim 1,
wherein the prior map is linked to the three-dimensional model of the environment by: matching a plurality of registration features of the prior map with corresponding registration features of the three-dimensional model; and registering the prior map in alignment with the three-dimensional model based on the matched plurality of registration features.( Lee teaches see para 0030-0031 and 0053 [0053] The identification of the spatial features in an image provides the relative location of those spatial features in a two-dimensional space, that is, “2D spatial features.” In order to map a 2D spatial feature to a third dimension (i.e., the distance, or “depth” from the electronic device 100), that is, to determine the corresponding “3D image feature”, the electronic device 100 determines the depth of the 2D feature relative to the electronic device 100 using one or both of Multiview image analysis or analysis using the depth sensor data. )
Regarding claim 4, The computer-implemented method of claim 1,
wherein the imaging data comprises one or more of:
Lidar data obtained by a Lidar of the portable imaging apparatus; and
visual data obtained by a visual camera of the portable imaging apparatus. Lee teaches:(see para[0053] The identification of the spatial features in an image provides the relative location of those spatial features in a two-dimensional space, that is, “2D spatial features.” In order to map a 2D spatial feature to a third dimension (i.e., the distance, or “depth” from the electronic device 100), that is, to determine the corresponding “3D image feature”, the electronic device 100 determines the depth of the 2D feature relative to the electronic device 100 using one or both of multiview image analysis or analysis using the depth sensor data.)
Regarding claim 6, The computer-implemented method of claim 1,
wherein the method further comprises:
receiving survey imaging data indicative of the environment, the survey imaging data captured by the portable imaging apparatus located within the environment; and creating the prior map of the environment using the survey imaging data.
Lee teaches (see para 0033] Another example application of the relative position/orientation determination process can include, for example, missing/new object detection whereby the appearance of a new object or the disappearance of a previously identified object can be determined based on a comparison of the expected local environment view of the electronic device 100 for a given relative position and orientation to the actual local environment view captured by the electronic device 100 in the same position/orientation. As described below, the geometric uncertainty introduced by differences between an expected environment and the actual encountered environment can trigger various operations, including a refresh operation whereby the electronic device 100 initiates a remapping of the portion of the local environment 112 exhibiting the change. See also para 0031 and 0058)
Regarding claim 7, The computer-implemented method of claim 6,
wherein creating the prior map of the environment using the survey imaging data comprises: creating a plurality of prior sub-maps using the survey imaging data, each prior sub-map of the plurality of sub-maps indicative of a respective portion of the environment and assembling the plurality of sub-maps together to form the prior map. Lee teaches [0033] Another example application of the relative position/orientation determination process can include, for example, missing/new object detection whereby the appearance of a new object or the disappearance of a previously identified object can be determined based on a comparison of the expected local environment view of the electronic device 100 for a given relative position and orientation to the actual local environment view captured by the electronic device 100 in the same position/orientation. As described below, the geometric uncertainty introduced by differences between an expected environment and the actual encountered environment can trigger various operations, including a refresh operation whereby the electronic device 100 initiates a remapping of the portion of the local environment 112 exhibiting the change. See also para 0031))
Regarding claim 8, The computer-implemented method of claim 7,
wherein the method further comprises:
receiving update survey imaging data indicative of the respective portion of the environment represented by a corresponding prior sub-map of the plurality of prior sub-maps; creating an updated prior-sub map using the update survey imaging data; and replacing the corresponding prior sub-map of the plurality of prior sub-maps using the updated prior-sub map.
Lee teaches ( see para 0030…. As the electronic device 100 moves relative to the previous view, the electronic device 100 updates the graphical overlay so as to reflect the changed perspective. Moreover, the head tracking data 140 can be used to detect changes in the position of the head 122 of the user 110 relative to the display 108, in response to which the electronic device 100 can adjust the displayed graphical representation 160 so as to reflect the changed viewing angle of the user 110 relative to the display 108. See also para 0058… The electronic device 100 can utilize these same SLAM techniques using multiple iterations of the position/orientation determination process of block 716 over time so as to generate a map of the local environment 112 while concurrently determining and updating the position/orientation of the electronic device 100 at each appropriate point in time. This local mapping information can be utilized by the electronic device 100 to support any of a variety of location-based functionality, such as use in determining a path for a user to a specified destination and providing visual navigational aids to the user according to this path, as described in greater detail below.)
Regarding claim 9. The computer-implemented method of claim 8,
comprising: matching a plurality of registration features of the updated prior sub-map with corresponding registration features of the corresponding prior sub-map; and registering the updated prior sub-map in alignment with the corresponding prior sub-map based on the matched plurality of registration features prior to replacing the corresponding prior sub-map.
Lee teaches (See para 0065… As the user (and the electronic device 100) moves along the path to the exit door, the electronic device 100 can update the navigational arrow graphic presented to reflect any changes in direction necessary to continue navigating the path to the exit door. In a more sophisticated application, electrical wiring and HVAC duct location information for the office may be stored in a computer-aided drawing (CAD) form such that the electronic device 100 can present the graphical representations of the electrical wiring and HVAC duct locations present in the presented image frame of the area of the office facing the rear of the electronic device 100 in a three-dimensional form that correlates to the relative positions/orientations of the corresponding walls, floors, and ceilings present in the presented image. As the user moves the electronic device 100 around the office, the presented image of the local environment 112 changes and thus the electronic device 100 updates the electrical wiring and HVAC duct overlay to reflect the changes in the area of the office presented as imagery at the display 108.)
Regarding claim 10. The computer-implemented method of claim 1,
wherein the overlay information comprises at least one interaction region overlaying a displayed feature in the three-dimensional model, and wherein the method further comprises: receiving a user read input to select the interaction region; and in response to the user read input, retrieving stored data relating to the displayed feature from the overlay information and outputting the retrieved stored data.
Lee teaches (see para 0066] The view perspective presented by the graphical overlay also may be modified based on changes in the position of the user's head (or the user's eyes) relative to the display 108. To this end, the electronic device 100 can react to head/eye position changes as represented in the head tracking or eye tracking information captured at block 708 to change the view perspective of the image and graphical overlay presented at the display 108 and para 0095…. The current context of the electronic device 100 also may be used in determining the appropriate activation configuration. To illustrate, if the current context indicates that the user is using the electronic device 100 to provide an AR graphical overlay that is supposed to precisely identify the location of non-visible or buried objects, it may be more imperative that the electronic device 100 accurately identify the relative 3D positions of spatial features so as to accurately position the AR graphical overlay over the underlying captured image.)
Regarding claim 11.The computer-implemented method of claim 1,
wherein the overlay information comprises at least one interaction region overlaying a displayed feature in the three-dimensional model, and wherein the method further comprises: receiving a user write input associated with the interaction region, the user write input representative of data to store in relation to the displayed feature; and causing the data to be stored in the overlay information in relation to the displayed feature.
Lee teaches (see para 0066] The view perspective presented by the graphical overlay also may be modified based on changes in the position of the user's head (or the user's eyes) relative to the display 108. To this end, the electronic device 100 can react to head/eye position changes as represented in the head tracking or eye tracking information captured at block 708 to change the view perspective of the image and graphical overlay presented at the display 108 and para 0095[0095] The current context of the electronic device 100 also may be used in determining the appropriate activation configuration. To illustrate, if the current context indicates that the user is using the electronic device 100 to provide an AR graphical overlay that is supposed to precisely identify the location of non-visible or buried objects, it may be more imperative that the electronic device 100 accurately identify the relative 3D positions of spatial features so as to accurately position the AR graphical overlay over the underlying captured image.)
Regarding claim 12. The computer-implemented method of claim 11,
wherein the data to store in relation to the displayed feature in the three-dimensional model comprises one or more of:
an observation associated with the displayed feature; and
a measurement associated with the displayed feature and recorded using the portable imaging apparatus at the identified location of the portable imaging apparatus.
Lee teaches (see para 0031 ….The electronic device 100 then can generate a graphical overlay with visual representations of the electrical wiring positioned and oriented relative to corresponding spatial features (e.g., the corners 124, 126, and 128) identified in the video imagery. As illustrated in FIG. 1, the graphical overlay can include colored dashed lines 152 and 154 representing electrical wiring in the current view and description balloons 156 and 158 to provide descriptions of the electrical wiring, such as wiring type, an identifier associated with the wiring, and the building components powered by the corresponding wiring. The electronic device 100 then jointly presents the graphical overlay and the video imagery at the display 108 so as to present the user 110 with a graphical representation 160 of the location of electrical wiring within the current view of the local environment 112 as captured by the narrow angle imaging camera 116.
Regarding claim 13. The computer-implemented method of claim 1,
wherein: receiving the imaging data indicative of the environment comprises receiving periodically updated imaging data at an imaging update rate, the imaging data captured by the portable imaging apparatus located at a moving position within the environment; identifying the location of the portable imaging apparatus in the environment comprises matching the identified object feature in the imaging data with the corresponding object feature in the prior map of the environment at a matching update rate, wherein the imaging update rate is higher than the matching update rate; and outputting the overlay information with the displayed representation of the environment comprises outputting the overlay information with the displayed representation of the environment viewed from the identified moving position of the portable imaging apparatus.
Lee teaches ( See 0067] As noted above, the electronic device 100 cycles through iterations of the method 700 to provide real-time, updated localization, mapping, and augmented reality display. However, these sub-processes do not necessarily cycle at the same rate. To illustrate, the image alignment and AR processes may update/cycle at the same frame rate as the imaging cameras 114, 116, and 118 because these processes are directly tied to the captured imagery. However, the non-image sensor capture and current context determination may proceed at different cycle rates. To illustrate, it may be appropriate to capture gyroscopic or inertial sensor states more frequently than the frame rate in order to have sufficiently accurate inertial navigation estimation. Conversely, the location-related features of the electronic device 100 may not require a high position resolution, and thus the image analysis process to determine the current position/orientation of the electronic device 100 may occur at a cycle rate slower than the frame rate of the imaging cameras. )
Regarding claim 14. The computer-implemented method of claim 1,
wherein receiving the imaging data, identifying the location of the portable imaging apparatus, and outputting the overlay information with the displayed representation of the environment viewed from the identified location of the portable imaging apparatus are performed at the portable imaging apparatus.
Lee teaches (see para [0030-0031] Moreover, the relative position/orientation information obtained by the electronic device 100 can be combined with supplemental information 144 to present an augmented reality (AR) view of the local environment 112 to the user 110 via the display 108 of the electronic device 100. This supplemental information 144 can include one or more AR databases locally stored at the electronic device 100 or remotely accessible by the electronic device 100 via a wired or wireless network. And para 0031.. Accordingly, the electronic device 100 can capture video imagery of a view of the local environment 112 via the imaging camera 116, determine a relative orientation/position of the electronic device 100 as described above and herein, and determine the position and orientation of electrical wiring located within the walls present in the view of the local environment. The electronic device 100 then can generate a graphical overlay with visual representations of the electrical wiring positioned and oriented relative to corresponding spatial features (e.g., the corners 124, 126, and 128) identified in the video imagery. As illustrated in FIG. 1, the graphical overlay can include colored dashed lines 152 and 154 representing electrical wiring in the current view and description balloons 156 and 158 to provide descriptions of the electrical wiring, such as wiring type, an identifier associated with the wiring, and the building components powered by the corresponding wiring. The electronic device 100 then jointly presents the graphical overlay and the video imagery at the display 108 so as to present the user 110 with a graphical representation 160 of the location of electrical wiring within the current view of the local environment 112 as captured by the narrow angle imaging camera 116. As the electronic device 100 moves relative to the previous view, the electronic device 100 updates the graphical overlay so as to reflect the changed perspective. Moreover, the head tracking data 140 can be used to detect changes in the position of the head 122 of the user 110 relative to the display 108, in response to which the electronic device 100 can adjust the displayed graphical representation 160 so as to reflect the changed viewing angle of the user 110 relative to the display 108..)
Regarding claim 15. The computer-implemented method of claim 1,
wherein outputting the overlay information of the environment with the displayed representation of the environment viewed from the identified location of the portable imaging apparatus comprises outputting the overlay information overlaying a corresponding portion of the three-dimensional model or a corresponding portion of the imaging data, to provide an augmented reality image of the environment.
Lee teaches ( para 0031.. Accordingly, the electronic device 100 can capture video imagery of a view of the local environment 112 via the imaging camera 116, determine a relative orientation/position of the electronic device 100 as described above and herein, and determine the position and orientation of electrical wiring located within the walls present in the view of the local environment. The electronic device 100 then can generate a graphical overlay with visual representations of the electrical wiring positioned and oriented relative to corresponding spatial features (e.g., the corners 124, 126, and 128) identified in the video imagery. As illustrated in FIG. 1, the graphical overlay can include colored dashed lines 152 and 154 representing electrical wiring in the current view and description balloons 156 and 158 to provide descriptions of the electrical wiring, such as wiring type, an identifier associated with the wiring, and the building components powered by the corresponding wiring. The electronic device 100 then jointly presents the graphical overlay and the video imagery at the display 108 so as to present the user 110 with a graphical representation 160 of the location of electrical wiring within the current view of the local environment 112 as captured by the narrow angle imaging camera 116. As the electronic device 100 moves relative to the previous view, the electronic device 100 updates the graphical overlay so as to reflect the changed perspective. Moreover, the head tracking data 140 can be used to detect changes in the position of the head 122 of the user 110 relative to the display 108, in response to which the electronic device 100 can adjust the displayed graphical representation 160 so as to reflect the changed viewing angle of the user 110 relative to the display 108..) )
Regarding claim 16. The computer-implemented method of claim 1,
wherein outputting the overlay information with the displayed representation of the environment comprises providing output signaling indicative of the overlay information with the displayed representation of the environment viewed from the identified location of the portable imaging apparatus to one or more of: a display screen of the portable imaging apparatus; and a display screen remote from the portable imaging apparatus.
Lee teaches [0030] Moreover, the relative position/orientation information obtained by the electronic device 100 can be combined with supplemental information 144 to present an augmented reality (AR) view of the local environment 112 to the user 110 via the display 108 of the electronic device 100. This supplemental information 144 can include one or more AR databases locally stored at the electronic device 100 or remotely accessible by the electronic device 100 via a wired or wireless network. )
Regarding claim 17. The computer-implemented method of claim 1,
wherein the environment is an indoor environment.
Lee teaches ( As the user moves the electronic device 100 around the office, the presented image of the local environment 112 changes and thus the electronic device 100 updates the electrical wiring and HVAC duct overlay to reflect the changes in the area of the office presented as imagery at the display 108. )
Regarding claims 18 and 22 they are rejected similar to claim 1 above and see rejection of claim 1 above..
Regarding claim 19, it is rejected similar to claim 4.See rejection of claim 4 above.
Regarding claim 21, The portable imaging apparatus of claim 19,
further comprising input means configured to receive one of more of:
a user read input, the user read input configured to select an interaction region corresponding to a displayed feature in the three-dimensional model, retrieve stored data relating to the displayed feature from the overlay information and output the retrieved stored data; and
a user write input, the user write input associated with the interaction region and representative of data to store in relation to the displayed feature[[;]] and configured to cause the data to be stored in the overlay information in relation to the displayed feature.
Lee teaches see para 0066 The view perspective presented by the graphical overlay also may be modified based on changes in the position of the user's head (or the user's eyes) relative to the display 108. To this end, the electronic device 100 can react to head/eye position changes as represented in the head tracking or eye tracking information captured at block 708 to change the view perspective of the image and graphical overlay presented at the display 108 and para 0095…. The current context of the electronic device 100 also may be used in determining the appropriate activation configuration. To illustrate, if the current context indicates that the user is using the electronic device 100 to provide an AR graphical overlay that is supposed to precisely identify the location of non-visible or buried objects, it may be more imperative that the electronic device 100 accurately identify the relative 3D positions of spatial features so as to accurately position the AR graphical overlay over the underlying captured image.)
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4,8,14-19 and 22 are rejected under 35 U.S.C. 102(a)(1) being anticipated by Dedonato et al (2021/0150818A1).
Regarding claim 1, A computer-implemented method of localization of a portable imaging apparatus in an environment, the method comprising: Dedonato teaches: See abstract The AR device can capture images of the user's physical environment to generate or identify a user's location.
receiving imaging data indicative of the environment, the imaging data captured by the portable imaging apparatus located at a position within the environment;
Dedonato teaches: [0109] FIG. 8 is a process flow diagram of an example of a method 800 of rendering virtual content in relation to recognized objects. The method 800 describes how a virtual scene may be presented to a user of the wearable system …See also abstract The AR device can project graphics at designated locations within the user's environment to guide the user to capture images of the user's physical environment.
identifying a location of the portable imaging apparatus in the environment by matching an identified object feature in the imaging data with a corresponding object feature in a prior map of the environment, the prior map linked to a three-dimensional model of the environment; Dedonato The object recognizers 708a-708n may crawl through these collected points and recognize one or more objects using a map database at block 830. This information may then be conveyed to the user's individual wearable system at block 840, and the desired virtual scene may be accordingly displayed to the user at block…. A pose process 910 may be executed on the wearable computing architecture (e.g., processing module 260 or controller 460) and utilize data from the map 920 to determine position and orientation of the wearable computing hardware or user. Pose data may be computed from data collected on the fly as the user is experiencing the system and operating in the world. The data may comprise images, data from sensors (such as inertial measurement units, which generally comprise accelerometer and gyroscope components) and surface information pertinent to objects in the real or virtual environment.
outputting overlay information with a displayed representation of the environment viewed from the identified location of the portable imaging apparatus, the overlay information retrieved from the prior map and indicative of prior stored data relating to a three-dimensional model feature in the three-dimensional model. Dedonato: This information may then be conveyed to the user's individual wearable system at block 840, and the desired virtual scene may be accordingly displayed to the user at block 850. For example, the desired virtual scene (e.g., user in CA) may be displayed at the appropriate orientation, position, etc., in relation to the various objects and other surroundings of the user in New York…. See also para 0007 collect data associated with the environment of the user; in response to collecting sufficient data associated with the unmapped location, signal to the user that data was successfully collected; and in response to determining that at least one marker within the collected data correspond to at least one marker in the map, displaying virtual reality content associated with the map;
Regarding claim 2, The computer-implemented method of claim 1,
wherein the method further comprises: identifying a view direction of the portable imaging apparatus in the environment by matching the identified object feature in the imaging data with the corresponding object feature in [[a]] the prior map of the environment, and wherein outputting the overlay information with the displayed representation of the environment comprises outputting the overlay information with the displayed representation of the environment viewed from the identified location of the portable imaging apparatus in the identified view direction.
Dedonato teaches: The user device can constantly update information about the user's environment and receive information about the world map 1210. The world map 1210 may be created by the user or by someone else. As discussed herein, user devices (e.g. 1230a, 1230b, 1230c) and remote computing system 1220, alone or in combination, may construct or update the world map 1210. For example, a user device may be in communication with the remote processing module 1270 and the remote data repository 1280. The user device may acquire or process information about the user and the user's environment.
Regarding claim 3, The computer-implemented method of claim 1,
wherein the prior map is linked to the three-dimensional model of the environment by: matching a plurality of registration features of the prior map with corresponding registration features of the three-dimensional model; and registering the prior map in alignment with the three-dimensional model based on the matched plurality of registration features.
Dedonato teaches: See para 0009.. [0009] In some examples, an augmented reality (AR) system can include: an AR display configured to present virtual content to a user of the AR system; an outward facing camera configured to capture one or more images of a three-dimensional (3D) environment of the user; and a hardware processor in communication with the AR display and the outward facing camera, the hardware processor can be programmed to: receive an invitation to access a shared map; facilitate user access to the shared map; guide the user to observe the 3D environment of the user; collect data associated with the 3D environment of the user; and load content associated with the shared map if data associated with the shared map is found in the 3D environment of the user.
Regarding claim 4, The computer-implemented method of claim 1,
wherein the imaging data comprises one or more of:
Lidar data obtained by a Lidar of the portable imaging apparatus; and
visual data obtained by a visual camera of the portable imaging apparatus.
Dedonato teaches: Para 0009 an outward facing camera configured to capture one or more images of a three-dimensional (3D) environment of the user;
Regarding claim 8, The computer-implemented method of claim 7,
wherein the method further comprises:
receiving update survey imaging data indicative of the respective portion of the environment represented by a corresponding prior sub-map of the plurality of prior sub-maps; creating an updated prior-sub map using the update survey imaging data; and replacing the corresponding prior sub-map of the plurality of prior sub-maps using the updated prior-sub map.
Dedonato teaches: (0133] One or more of the user devices can be used with the user input device 466 shown in FIG. 4. A user device can obtain information about the user and the user's environment (e.g., using the outward-facing imaging system 464 shown in FIG. 4). The user device or remote computing system 1220 can construct, update, and build a collection of images, points and other information using the information obtained from the user devices…. [0135] The remote data repository 1280 can be used to store data and to facilitate the construction of the world map 1210. The user device can constantly update information about the user's environment and receive information about the world map 1210.
Regarding claim 14. The computer-implemented method of claim 1,
wherein receiving the imaging data, identifying the location of the portable imaging apparatus, and outputting the overlay information with the displayed representation of the environment viewed from the identified location of the portable imaging apparatus are performed at the portable imaging apparatus.
Dedonato teaches: [0020] FIG. 8 is a process flow diagram of an example of a method of rendering virtual content in relation to recognized objects.
Regarding claim 15. The computer-implemented method of claim 1,
wherein outputting the overlay information of the environment with the displayed representation of the environment viewed from the identified location of the portable imaging apparatus comprises outputting the overlay information overlaying a corresponding portion of the three-dimensional model or a corresponding portion of the imaging data, to provide an augmented reality image of the environment.
Dedonato teaches: [0388] Example 43: An augmented reality (AR) system comprising: [0389] an AR display configured to present virtual content to a user of the AR system; [0390] an outward facing camera configured to capture one or more images of a three-dimensional (3D) environment of the user; and [0391] a hardware processor in communication with the AR display and the outward facing camera, the hardware processor programmed to: [0392] receive an invitation to access a shared map; [0393] facilitate user access to the shared map; [0394] guide the user to observe the 3D environment of the user; [0395] collect data associated with the 3D environment of the user; and [0396] load content associated with the shared map if data associated with the shared map is found in the 3D environment of the user.
Regarding claim 16. The computer-implemented method of claim 1,
wherein outputting the overlay information with the displayed representation of the environment comprises providing output signaling indicative of the overlay information with the displayed representation of the environment viewed from the identified location of the portable imaging apparatus to one or more of:
a display screen of the portable imaging apparatus; and a display screen remote from the portable imaging apparatus.
Dedonato teaches: [0020] FIG. 8 is a process flow diagram of an example of a method of rendering virtual content in relation to recognized objects….This information may then be conveyed to the user's individual wearable system at block 840, and the desired virtual scene may be accordingly displayed to the user at block 850
Regarding claim 17. The computer-implemented method of claim 1,
wherein the environment is an indoor environment.-
Dedonato teaches: [0136] FIG. 13A illustrates a user 1331 wearing an AR display system rendering AR content as the user 1331 moves through a physical world environment 1333.
Regarding claims 18 and 22 they are rejected similar to claim 1 above and see rejection of claim 1 above…
Regarding claim 19, it is rejected similar to claim 4.See rejection of claim 4 above.
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4,14-19 and 22 are rejected under 35 U.S.C. 102(a)(1) being anticipated by Kahle et al. (2022/035873 A1).
Regarding claim 1,A computer-implemented method of localization of a portable imaging apparatus in an environment, the method comprising:
receiving imaging data indicative of the environment, the imaging data captured by the portable imaging apparatus located at a position within the environment;
Kahle teaches: Para 005 The augmented-reality device is configured to provide augmented-reality to a user. The augmented-reality device comprises one or more sensors configured to acquire images of the environment; …see also Para [0029] The AR device 104 is configured to provide augmented-reality to a user 110. The AR device 104 comprises one or more sensors (e.g., one or more cameras and/or LIDAR) configured to acquire images of the environment
identifying a location of the portable imaging apparatus in the environment by matching an identified object feature in the imaging data with a corresponding object feature in a prior map of the environment, the prior map linked to a three-dimensional model of the environment; Para 005 a display configured to render graphics on the display; and/or one or more processors configured to: acquire, using the laser scanner, a collection of data points representing a point cloud, wherein the point cloud is a three-dimensional point cloud; transmit data of the point cloud to the augmented-reality device; align a reference frame of the augmented-reality device to the data of the point cloud; (see also para 0030) align a reference frame of the augmented-reality device to the data of the point cloud; and
outputting overlay information with a displayed representation of the environment viewed from the identified location of the portable imaging apparatus, the overlay information retrieved from the prior map and indicative of prior stored data relating to a three-dimensional model feature in the three-dimensional model. Para 005 and/or present a graphic on the display of the augmented-reality device, wherein the graphic is positioned on the display in relation to the environment based on the reference frame of the augmented-reality device being aligned to data of the point cloud.(see also para 0030 present a graphic on the display of the augmented-reality device, wherein the graphic is positioned on the display in relation to the environment based on the reference frame of the augmented-reality device being aligned to data of the point cloud. (See also Para. 0035] FIG. 3 is an embodiment of a 3D overlay in an augmented-reality (AR) device. FIG. 3 depicts a display 304 of an AR device. A graphic 308 is shown as an overlay in the display 304 of the AR device. The graphic is generated from model data of a three-dimensional model (e.g., a CAD drawing). The model data is aligned with the reference frame of the AR device, so that the graphic 308 is depicted in relation to the environment. For example, the graphic 308 is an image of a water pipe or electrical line to be placed at a construction site. In some embodiments, the AR device can enable the user can “see” through walls or obstacles (e.g., to see wire locations in a wall).
Regarding claim 2, The computer-implemented method of claim 1,
wherein the method further comprises: identifying a view direction of the portable imaging apparatus in the environment by matching the identified object feature in the imaging data with the corresponding object feature in [[a]] the prior map of the environment, and wherein outputting the overlay information with the displayed representation of the environment comprises outputting the overlay information with the displayed representation of the environment viewed from the identified location of the portable imaging apparatus in the identified view direction.
Kahle teaches: see para 0030 present a graphic on the display of the augmented-reality device, wherein the graphic is positioned on the display in relation to the environment based on the reference frame of the augmented-reality device being aligned to data of the point cloud. (See also Para. 0035] FIG. 3 is an embodiment of a 3D overlay in an augmented-reality (AR) device. FIG. 3 depicts a display 304 of an AR device. A graphic 308 is shown as an overlay in the display 304 of the AR device. The graphic is generated from model data of a three-dimensional model (e.g., a CAD drawing). The model data is aligned with the reference frame of the AR device, so that the graphic 308 is depicted in relation to the environment. For example, the graphic 308 is an image of a water pipe or electrical line to be placed at a construction site. In some embodiments, the AR device can enable the user can “see” through walls or obstacles (e.g., to see wire locations in a wall).
Regarding claim 3, The computer-implemented method of claim 1,
wherein the prior map is linked to the three-dimensional model of the environment by: matching a plurality of registration features of the prior map with corresponding registration features of the three-dimensional model; and registering the prior map in alignment with the three-dimensional model based on the matched plurality of registration features.
Kahle teaches: (See also Para. 0035] FIG. 3 is an embodiment of a 3D overlay in an augmented-reality (AR) device. FIG. 3 depicts a display 304 of an AR device. A graphic 308 is shown as an overlay in the display 304 of the AR device. The graphic is generated from model data of a three-dimensional model (e.g., a CAD drawing). The model data is aligned with the reference frame of the AR device, so that the graphic 308 is depicted in relation to the environment. For example, the graphic 308 is an image of a water pipe or electrical line to be placed at a construction site. In some embodiments, the AR device can enable the user can “see” through walls or obstacles (e.g., to see wire locations in a wall).
Regarding claim 4, The computer-implemented method of claim 1,
wherein the imaging data comprises one or more of:
Lidar data obtained by a Lidar of the portable imaging apparatus; and
visual data obtained by a visual camera of the portable imaging apparatus.
Kahle teaches: (0029] The AR device 104 is configured to provide augmented-reality to a user 110. The AR device 104 comprises one or more sensors (e.g., one or more cameras and/or LIDAR) configured to acquire images of the environment.
Regarding claim 14. The computer-implemented method of claim 1,
wherein receiving the imaging data, identifying the location of the portable imaging apparatus, and outputting the overlay information with the displayed representation of the environment viewed from the identified location of the portable imaging apparatus are performed at the portable imaging apparatus.
Kahle teaches: [0029] The AR device 104 is configured to provide augmented-reality to a user 110. The AR device 104 comprises one or more sensors (e.g., one or more cameras and/or LIDAR) configured to acquire images of the environment and a display configured to render graphics on the display. The AR device 104 has a field-of-view 116
Regarding claim 15. The computer-implemented method of claim 1,
wherein outputting the overlay information of the environment with the displayed representation of the environment viewed from the identified location of the portable imaging apparatus comprises outputting the overlay information overlaying a corresponding portion of the three-dimensional model or a corresponding portion of the imaging data, to provide an augmented reality image of the environment.
Kahle teaches:( Para 0035] FIG. 3 is an embodiment of a 3D overlay in an augmented-reality (AR) device. FIG. 3 depicts a display 304 of an AR device. A graphic 308 is shown as an overlay in the display 304 of the AR device. The graphic is generated from model data of a three-dimensional model (e.g., a CAD drawing). The model data is aligned with the reference frame of the AR device, so that the graphic 308 is depicted in relation to the environment. For example, the graphic 308 is an image of a water pipe or electrical line to be placed at a construction site
Regarding claim 16. The computer-implemented method of claim 1,
wherein outputting the overlay information with the displayed representation of the environment comprises providing output signaling indicative of the overlay information with the displayed representation of the environment viewed from the identified location of the portable imaging apparatus to one or more of: a display screen of the portable imaging apparatus; and a display screen remote from the portable imaging apparatus.
Kahle teaches: See also Para. 0035] FIG. 3 is an embodiment of a 3D overlay in an augmented-reality (AR) device. FIG. 3 depicts a display 304 of an AR device. A graphic 308 is shown as an overlay in the display 304 of the AR device. The graphic is generated from model data of a three-dimensional model (e.g., a CAD drawing). The model data is aligned with the reference frame of the AR device, so that the graphic 308 is depicted in relation to the environment. For example, the graphic 308 is an image of a water pipe or electrical line to be placed at a construction site. In some embodiments, the AR device can enable the user can “see” through walls or obstacles (e.g., to see wire locations in a wall).
Regarding claim 17. The computer-implemented method of claim 1,
wherein the environment is an indoor environment.- The model data is aligned with the reference frame of the AR device, so that the graphic 308 is depicted in relation to the environment. For example, the graphic 308 is an image of a water pipe or electrical line to be placed at a construction site.
Kahle teaches:
Regarding claims 18 and 22 they are rejected similar to claim 1 above and see rejection of claim 1 above…
Regarding claim 19, it is rejected similar to claim 4.See rejection of claim 4 above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TEMESGHEN GHEBRETINSAE whose telephone number is (571)272-3017. The examiner can normally be reached Monday-Friday 7:30-4.00 p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, TEMESGHEN GHEBRETINSAE can be reached at 571-272-3017. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TEMESGHEN GHEBRETINSAE/Supervisory Patent Examiner, Art Unit 2626 8/8/26
Prosecution Insights