Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 15 recites “a computer-readable medium comprising instructions that upon execution by the processor…” The examiner suggests the following amendment: “a computer-readable medium comprising instructions that upon execution by the one or more processors…”
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the claims at issue are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the reference application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO internet Web site contains terminal disclaimer forms which may be used. Please visit http://www.uspto.gov/forms/. The filing date of the application will determine what form should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to http://www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1-20 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. 11,348,316 B2 (“the '316 patent”) in view of Mak (Pat. No. US 11,645,034; “Mak” hereinafter). Although the conflicting claims are not identical, they are not patentably distinct from each other because the instant claims recite an obvious variant of the subject matter claimed in the '316 patent, namely the extension of the '316 patent's single-virtual-element operation to a plurality of virtual elements provided by separately executing applications, a configuration that Mak discloses and contemplates.
Regarding claim 1, claim 1 of the '316 patent discloses a method comprising, at a device with a processor and a computer readable medium: providing a view comprising a virtual element and a physical environment, wherein the virtual element is a first type of modality (i.e., a first set of interactive features) and is at a first location in the view relative to the physical environment; detecting a change in the view comprising placement of the virtual element at a second location in the view; detecting an attribute of the physical environment at the second location; selecting a second type of modality (i.e., a second set of interactive features) of the virtual element based on the detected attribute; and, in accordance with a determination to update the virtual element, updating the virtual element by changing a size, shape, design, or color attribute of the virtual element and changing the modality-specific functions of the virtual element from the first set of interactive features to the second set of interactive features, and presenting an updated view of the updated virtual element and the physical environment. The instant claim 1 recites substantially the same method, differing only in that it recites a plurality of virtual elements provided from a plurality of individual, separately executing applications, rather than a single virtual element, undergoing the identical sequence of operations: providing a view of a plurality of virtual elements each having a set of interactive features and a physical environment; detecting a change comprising placement of one of the virtual elements at a changed location; detecting an attribute of the physical environment at the changed location; changing the set of interactive features of that virtual element to a different set based on the detected attribute; based on the different set of interactive features and the changed location, changing at least one of a size, a shape, a design, and a color attribute of the virtual element; and presenting an updated view.
Mak teaches providing a view comprising a plurality of virtual elements from a plurality of individual, separately executing applications and a representation of a physical environment (see the rejection of the instant claim 1 below).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the instant application to extend the single-virtual-element method of claim 1 of the ‘316 patent to a plurality of virtual elements from a plurality of individual, separately executing applications, as taught by Mak, because doing so amounts to nothing more than the routine repetition of a known operation across multiple instances of the same claimed element, an obvious variation that does not render the instant claim patentably distinct from the '316 patent’s claim 1.
Regarding claims 2-14, these claims recite limitations that are not patentably distinct from corresponding dependent claims of the '316 patent applied to the plurality of virtual elements of claim 1 in the same obvious manner discussed above: claim 5 (obtaining an image via an image sensor and generating a representation of the physical environment) corresponds to '316 claim 2; claim 6 (presenting a virtual element on an optical see-through display responsive to identifying a feature) corresponds to '316 claim 3; claim 7 (updating to a first or second visual appearance based on the attribute) corresponds to '316 claim 4; claim 8 (updating to a first or second functionality based on the attribute) corresponds to '316 claim 5; claim 9 (detecting the change via user input) corresponds to '316 claim 6; claims 10-12 (determining that the virtual element is on a planar, horizontal planar, or vertical planar surface) correspond to '316 claims 7-9; claim 13 (identifying a classification of a portion of the physical environment using a classifier and an amount of the portion available) corresponds to '316 claim 16; and claim 14 (displaying the updated virtual element at the changed location or away from it depending on the attribute) corresponds to '316 claim 17.
Regarding claims 15-19, claim 15 recites a system comprising an electronic device with a display and an image sensor, one or more processors, and a computer-readable medium comprising instructions to perform the same operations recited in claim 1, and is not patentably distinct from system claim 15 of the '316 patent for the same reasons discussed above with respect to claim 1. Claims 16-19 depend from claim 15 and recite limitations corresponding, respectively, to '316 claims 16 (classifier/amount of portion), 17 (first and second visual appearance), 18 (notification identifying the detected attribute sent to a first application), and 19 (the first application sending updated information in response to the notification), and are accordingly not patentably distinct from those claims for the same reasons.
Regarding claim 20, claim 20 recites a non-transitory computer-readable storage medium storing program instructions computer-executable to perform the operations recited in claim 1, and is not patentably distinct from claim 20 of the '316 patent, a corresponding storage-medium claim reciting the same operations, for the same reasons discussed above with respect to claim 1.
Claims 1-20 are further provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of U.S. Patent No. 12,211,152 B2 (“the '152 patent”) in view of Mak.
Regarding claim 1, claim 1 of the '152 patent discloses obtaining a request to display an application at a first location in a three-dimensional environment, wherein the application comprises a modality state definition for each of a plurality of positional states, each modality state definition comprising a different 2D or 3D model representation of the application having a different functionality and a different interactive feature; determining a first positional state associated with the first location; displaying, at the first location, a first 2D or 3D model representation of the application corresponding to the modality state definition matching the first positional state, with a visual representation of a first functionality and a first interactive feature; obtaining a second request to display the application at a second location; determining a second positional state associated with the second location; determining whether to update the first representation based on the second positional state; and, in accordance with a determination to update, displaying at the second location a second 2D or 3D model representation of the application corresponding to the modality state definition matching the second positional state, with a visual representation of a second functionality and second interactive feature different from the first. This is not patentably distinct from instant claim 1, which recites the same operative concept—detecting the placement of a virtual element (an application's on-screen representation) at a changed location, detecting an attribute of the physical environment at that location, and changing the element's interactive features and visual attributes (size, shape, design, or color) accordingly—applied, as in the analysis of the '316 patent above, to a plurality of virtual elements from separately executing applications rather than to a single application's representation.
Regarding claims 2-20, these claims recite limitations that are not patentably distinct from corresponding claims of the '152 patent applied to the plurality-of-applications variant of claim 1 in the same obvious manner discussed above: claims 5-6 (image sensor/optical see-through presentation) correspond to '152 claims 7-8; claim 9 (user input effecting the change) corresponds to '152 claim 9; claims 10-12 (planar, horizontal, and vertical planar surface determinations) correspond to '152 claims 10, 12, and 13; claim 13 (classifier-based identification of a portion of the environment) corresponds to '152 claim 2; claim 14 (displaying at or away from the changed location depending on the attribute) corresponds to '152 claim 5; claim 15 (system claim) corresponds to '152 claim 14; claims 16-19 correspond to '152 claims 15-19; and claim 20 (non-transitory computer-readable storage medium claim) corresponds to '152 claim 20.
The instant application, the '316 patent, and the '152 patent do not fall within one of the exceptions permitting patentably indistinct claims (e.g., 35 U.S.C. § 121), and applicant has not been given the benefit of a safe harbor provision. This provisional nonstatutory double patenting rejection is neither taken nor intended to be a final determination on the substance of the double patenting issue, but is a provisional rejection of the instant claims based on published/pending applications that have not yet issued as patents (to the extent the '152 patent is being cited as a pending or co-pending reference; where the '152 patent has already issued, the rejection set forth above is a non-provisional rejection). This rejection may be overcome by filing a terminal disclaimer in accordance with 37 C.F.R. § 1.321(c) or (d), disclaiming the terminal portion of the statutory term of any patent granted on the instant application that would extend beyond the expiration date of the full statutory term, as presently shortened by any terminal disclaimer, of the '316 patent and/or the '152 patent, and further providing that any patent granted on the instant application shall be enforceable only for and during such period that the instant patent and the reference patent(s) are commonly owned. A proper terminal disclaimer must be signed in compliance with 37 C.F.R. § 1.321(b).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Poulos et al. (US 2016/0155270, priority date May 13, 2013), in view of Mak (US 11,645,034, effectively filed Jun. 5, 2018).
Regarding claim 1, Poulos discloses a method comprising:
at a device with a processor and a computer readable medium (Poulos discloses a wearable augmented reality computing device comprising a see-through display device, a sensor system including one or more sensors, a logic subsystem, and a storage subsystem; see Poulos, Figs. 2-3 and the associated description):
providing a view comprising a (Poulos discloses providing, via a see-through display, a view of a virtual object (e.g. free-floating virtual object 130 in Fig. 1A) together with a representation of the user's physical environment; see Poulos, par. 21), wherein the (Poulos discloses that the free-floating virtual object can be grabbed and moved; see Poulos, pars. 36-37);
detecting a change in the view, the change comprising placement of (Poulos discloses that a user may input a speech command, such as “grab,” and then change a position of the user's focus, e.g., via head or eye movement, to move a virtual object from a first location to a new location; see Poulos, par. 38 and FIGS. 4-7);
detecting an attribute of the physical environment at the changed location in the view (Poulos discloses identifying a surface via image data acquired with the augmented reality computing device and determining attributes of that surface, including a type of surface, e.g., “wall” or “table,” surface transparency, surface texture, color, size, and orientation; see Poulos, pars. 19 and 26);
changing the set of interactive features of the virtual element that has a changed location, to a different set of interactive features based on the detected attribute (Poulos discloses that a free-floating virtual object can have different interactive features when it is attached to a surface. For example, it can change its orientation in response to a change in the user’s position (see par. 40). It can also slide across the surface in response to changes in the user’s gaze direction or position of focus (see par. 41). Furthermore, it can be pinned and unpinned to/from the surface in response to the user’s input (see pars. 42-43);
based on the different set of interactive features and the changed location, changing at least one of a size, a shape, a design, and a color attribute of the virtual element at the changed location (Poulos discloses that a virtual object attached to a surface may bend or deform to accommodate the shape of the surface; see Poulos, par. 46); and
presenting an updated view of the (In Fig. 1B of Poulos, the see-through display device 104 presents an updated view of the virtual TV 130 after it has been moved to the wall 118).
However, Poulos's disclosure is directed to a single virtual object, not a plurality of virtual objects from a plurality of individual, separately executing applications.
In the same field of augmented reality, Mak discloses providing a view comprising a plurality of virtual elements from a plurality of individual, separately executing applications together with a representation of a physical environment, including an example in which three separate TV applications (TV App1, TV App2, TV App3) each provide content elements displayed together relative to the physical environment (see Mak, col. 20, ll. 26-32, and FIG. 2E).
Therefore, it would have been obvious to a person skilled in the art before the effective filing date of the instant application to extend Poulos's policy-based, attribute-dependent interactive-feature and appearance determination, disclosed with respect to a single virtual object, to each of a plurality of virtual elements sourced from separately executing applications as disclosed by Mak, since Mak already discloses a shared augmented reality view populated by content from multiple, separately executing applications, and applying Poulos's known per-object policy mechanism to each such application-sourced element individually amounts to nothing more than the predictable use of a known technique to improve similar devices/environments in the same way, yielding the expected result of each virtual element's interactive features and appearance being appropriately and consistently tailored to the physical environment in which it is currently displayed.
Regarding claim 2, Poulos in view of Mak teaches the method of claim 1 wherein the device comprises an operating system including an application interface for exchanging information with the applications regarding the placement of virtual elements and selection of interactive features (Mak discloses that content elements are received from an entity, which may be an application (col. 11, ll. 46-48), and that the device's content structurization process and environment compositing process exchange information regarding the attributes of content elements and the selection of matched surfaces (col. 17, ll. 14-26).
Regarding claims 3 and 4, Poulos in view of Mak teaches the method of claim 1, further comprising:
sending a notification to a first application that provided a first virtual element, wherein the notification identifies the detected attribute; and sending, by the first application, information that updates a first set of interactive features to a second set of interactive features for the first virtual element that is different than the first set of interactive features based on the detected attribute in response to receiving the notification (Poulos discloses that a policy associated with a virtual object is consulted and applied whenever the object's surroundings change, such that the source associated with the virtual object effectively receives and responds to updated attribute information, and Mak similarly discloses that a content designer of a content element may explicitly indicate attributes controlling how the content element should be rendered, and that the system's matching and rendering modules apply the detected surface attributes to update the content element's presentation accordingly. Therefore, it would have been obvious to a PHOSITA to implement this exchange as sending a notification to a first application that provided a first virtual element, wherein the notification identifies the detected attribute, and to have the first application respond by sending... information that updates a first set of interactive features to a second set of interactive features based on the detected attribute, as this is merely the predictable implementation of the Poulos/Mak attribute-dependent matching architecture using conventional client-application callback or notification techniques, yielding no more than the expected result of allowing the content-providing application to control its own attribute-dependent presentation.
Regarding claim 5, Poulos in view of Mak discloses obtaining, using an image sensor, an image of the physical environment; generating the representation of the physical environment using the obtained image; and presenting the plurality of virtual elements with the representation of the physical environment (Poulos discloses acquiring image data of the physical environment using one or more image sensors of the augmented reality computing device, forming a surface reconstruction mesh representing the environment from that image data, and presenting virtual objects together with the environment via the see-through display (see Poulos, Description discussing image sensors 306 and surface reconstruction). Mak teaches presenting the plurality of virtual elements with the representation of the physical environment (Fig. 2E)).
Regarding claim 6, Poulos in view of Mak discloses identifying, using an image sensor, a feature in the physical environment; and presenting the plurality of virtual elements on an optical see-through display responsive to identifying the feature (Poulos discloses identifying, using an image sensor, a corner or discontinuity on a wall, and presenting a virtual object on a see-through display as being bent or deformed; see Fig. 9 and the associated description. Mak extends Poulos’s single virtual object to multiple virtual objects associated with multiple applications).
Regarding claim 7, Poulos in view of Mak discloses in accordance with determining that the attribute is a first attribute, updating the virtual element... to have a first visual appearance; and in accordance with determining that the attribute is a second attribute, updating the virtual element... to have a second visual appearance, wherein the first visual appearance is different from the second visual appearance (Poulos discloses modifying a virtual object's orientation and shape differently depending on whether the object attaches to a wall or a table surface, see Poulos, Description discussing FIGS. 4 and 9; Mak similarly discloses displaying content with a light-colored text/dark background scheme when matched to a surface in a dark real-world environment, and displaying the same content with a dark-colored text/light background scheme when matched to a surface in a bright real-world environment, see Mak, Description and FIGS. 3A-3B).
Regarding claim 8, Poulos in view of Mak discloses in accordance with determining that the attribute is a first attribute, updating the virtual element... to have a first functionality; and in accordance with determining that the attribute is a second attribute, updating the virtual element... to have a second functionality, wherein the first functionality is different from the second functionality (Poulos discloses that, depending on the detected type, texture, and color attributes of a candidate surface, a virtual object is permitted to attach to and be pinned at that surface (a first functionality) or is not permitted to attach to that surface (a second, different functionality, e.g., remaining a free-floating object that follows the user), see Poulos, Description discussing policies governing attachment and pin-ability, and the virtual chessboard/virtual poster examples).
Regarding claim 9, Poulos in view of Mak discloses detecting the change in the view comprises receiving user input representing a change in the location of the virtual element that has the changed location (Poulos discloses that a user may input a speech command, such as “grab,” and then change a position of the user's focus, e.g., via head or eye movement, to move a virtual object to a new location; see Poulos, par. 37).
Regarding claims 10-12, Poulos in view of Mak discloses determining that the virtual element that has the changed location is on a planar surface, on a horizontal planar surface, and on a vertical planar surface (Poulos expressly discloses displaying a virtual object as attached to a surface of a first orientation, e.g., a horizontal table surface, and, based on a policy, not displaying the virtual object as attached to a surface of a second, different orientation, e.g., a vertical wall surface, and vice versa; see Poulos, par. 27. Mak similarly discloses that surfaces to which content elements are matched can be vertical or horizontal; see Mak, col. 22, line 66 to col. 23, line 3).
Regarding claim 13, Poulos in view of Mak discloses identifying a classification of a portion of the physical environment using a classifier (Poulos discloses in par. 26 that: “attributes of real-world surfaces such as a type of surface, e.g., “wall” or “table,” surface transparency, e.g., translucent or opaque, surface texture, e.g., wood or glass, etc. may be determined”. This implies that a classifier is used in the determination process), and Mak discloses identifying an amount of the portion available upon which the virtual element that has the changed location can be positioned (Mak discloses determining an area attribute for each surface, representing the amount of the surface available for placement of a content element; see Mak, col. 17, ll. 14-26). It would have been obvious to a PHOSITA to combine Poulos's surface-classification technique with Mak's area-attribute determination for the reasons discussed above with respect to claim 1.
Regarding claim 14, Poulos in view of Mak discloses in accordance with determining that the attribute is of a first attribute, displaying the virtual element that has the changed location at a second location; and in accordance with determining that the attribute is of a second attribute, displaying the virtual element that has the changed location away from the second location (Poulos discloses that, when a policy indicates a candidate surface's attributes permit attachment, the virtual object is displayed as attached to that surface location, and that, when a policy indicates the attributes do not permit attachment, e.g., due to surface type or color mismatch, the virtual object is instead not displayed as attached to that surface and is displayed elsewhere, such as remaining a free-floating object away from that location; see Poulos, Description discussing policy-based attachment decisions; Mak similarly discloses that a disqualified candidate surface causes the system to instead render the content element on a newly created virtual surface offset from the original surface location, see Mak, Description discussing creating virtual objects module 144).
Regarding claims 15-19, these claims recite operations corresponding to those of claims 1, 8, 7, 3, and 4, respectively, applied to a system comprising: an electronic device with a display and an image sensor; one or more processors; and a computer-readable medium. Since Poulos also discloses such a system (See Figs. 2-3 and the associated description), these claims are unpatentable over Poulos in view of Mak for the same reasons discussed above with respect to those claims.
Regarding claim 20, Poulos disclose a non-transitory computer-readable storage medium storing program instructions that are computer-executable to perform operations corresponding to those recited in claim 1 above (Poulos discloses a storage subsystem holding instructions executable by a logic subsystem; see Poulos, pars. 64-65). Therefore, claim 20 is unpatentable over Poulos in view of Mak for the same reasons discussed above with respect to claim 1.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PHONG X NGUYEN whose telephone number is (571)270-1591. The examiner can normally be reached Mon-Fri 8am - 5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PHONG X NGUYEN/ Primary Patent Examiner, Art Unit 2617
Prosecution Insights