DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 8/15/2025 has been entered.
Terminal Disclaimer
Applicant has filed a Terminal Disclaimer filed 7/10/2024. The present application is subject to the statutory terms of U.S. Patent 10,627,625, U.S. Patent 10,921,599, USS. Patent 11, 287,659, and U.S. Patent 11,808,944. Response to Arguments
Response to Amendments
Claims 1-3, 5-9, 11-17, 21-23 have been amended to correct for the rejection made pursuant to 35 U.S.C. 112(b) and as such the rejection has been withdrawn.
Claim 25 has been canceled and as such all rejections have been withdrawn to the canceled claim as moot.
Response to Arguments
Applicant's arguments filed 12/19/2025 have been fully considered but they are not persuasive.
Regarding claim 1, applicant argues that the cited prior art of Shapira, Wheeler, Yamaguchi, Rolleston and Ooi fails to teach the combination of the amended claim language including: calculating a distance between an initial location of the selected item from the virtual menu of items and the location, determining whether the distance between distance is less than a first threshold distance, and rendering, when the distance is less than the first threshold distance, a orientation of the selected item to align one or more objects at the location by automatically orientating the selected item to cause a surface normal of the selected item to face the AR system, the surface normal being defined by the affordances of the selected item; automatically orientating the selected item to cause a surface normal of the selected item to be perpendicular to a surface of the location; or automatically orientating the selected item to cause a surface normal of the selected item to be parallel to a normal of the location (see pages 9-10 of applicant’s correspondence filed 12/19/2025). Examiner respectfully disagrees.
In response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., that all the features of the cited claim language are required) are not recited in the rejected claim(s). Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). In particular, claim 1 recites in the relevant limitations:
wherein, upon determining that the placing of the selected item at the location is suitable, the automatically placing the selected item at the location in the virtual environment of the user further includes:
(A) calculating a distance between an initial location of the selected item from the virtual menu of items and the location, determining whether the distance between distance is less than a first threshold distance, and rendering, when the distance is less than the first threshold distance, a orientation of the selected item to align one or more objects at the location by automatically orientating the selected item to cause a surface normal of the selected item to face the AR system, the surface normal being defined by the affordances of the selected item;
(B) automatically orientating the selected item to cause a surface normal of the selected item to be perpendicular to a surface of the location; or
(C) automatically orientating the selected item to cause a surface normal of the selected item to be parallel to a normal of the location
(emphasis and letter labels (A), (B), (C) added).
As the claim is currently drafted, the limitations are recited in the alternative such that only one of (A), (B), or (C) is required, and not all the claim limitations.
Shapira teaches the automatic orientation of a selected virtual item such as to cause a surface normal of the virtual item to be parallel to a normal of a location on which it is placed. In particular, Shapira teaches matching at least a virtual plane with a location plane such that the correspond (see Shapira, ¶72 disclosing geometric constraints ensures an object matches x,y,z coordinates; ¶¶151-158 disclosing reconstructing a scene as a 3D mesh or model representing the real-world environment, and models of virtual objects with transformations into the virtual space afforded by the corresponding real-word objects; ¶¶159-162 discloses constructing the TVR from the scene and room layout inputs by constructing a water-tight (i.e., enclosed space) representation of the environment, where the representation may be a set of planes, each labeled as floor, wall, ceiling, etc., all of which are guaranteed to be adjacent to another plane on all edges (therefore enclosed), and the scene layout interference algorithm includes ensuring that walls “stick” to observed walls (for tactile truth) – the lining up of planes in virtual coordinate space, where the physical constraints of the environment wall plane is also “virtual” as a mathematical representation of the physical environment, inherently results in an automatically orientating the selected item to cause a surface normal of the selected item to be parallel to a normal of the location as a principle of geometry).
Even if Shapira fails to teach the limitations, previously cited reference Ooi teaches wherein, upon determining that the placing of the selected item at the location is suitable, the automatically placing the selected item at the location in the virtual environment of the user further includes: (C) automatically orientating the selected item to cause a surface normal of the selected item to be parallel to a normal of the location (Ooi, ¶10: normal line of at least one face of virtual object becomes parallel with normal line of first face; Fig. 2 and ¶¶66-67: inserting virtual object into real space; ¶121: it is presumed that an instance of a virtual object is arranged such that it has the same tilt as a contacted face of a real object in the input image (that is, the two normal lines are parallel with each other))
Shapira, Wheeler, Rolleston and Ooi are directed to augmented reality systems for user interaction with objects. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, incorporating a menu for selection of the object and including the simulation of a force for placing a virtual object as further provided by Wheeler, further allowing a user to set an orientation speed of the object’s movement as provided by Yamaguchi and incorporating the use of repulsion for ensuring virtual objects are not misplaced based on other objects as taught by Rolleston, by further incorporating the technique for aligning virtual and physical objects as provided by Ooi, using known electronic interfacing and programming techniques. The modification results in an improved user interface for an augmented reality system by providing better visual alignment by coordinating the direction of objects to better line up for an improved visual presentation of virtual objects within a real-world scene, thus producing a more realistic and pleasing image.
Applicant’s remaining arguments are based on the same arguments for claim 1 and are therefore not persuasive for the same reasons above.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 15 is rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends. In particular, claim 1 (as currently amended) already recites the limitations recited by claim 15.
Accordingly, the claim merely recites duplicate limitations without more. Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, 7-9, 11-16, and 20-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Shapira et al. (US 2016/0253842 A1) in view of
Wheeler et al. (US 2013/0246967 A1) and
Yamaguchi et al. (US 2004/0164956 A1) in further view of
Rolleston et al. (US 2013/0155106 A1) and
Ooi (US 2012/0210255 A1).
Regarding claim 1, Shapira discloses:
An augmented reality (AR) system (Shapira, Abstract: An “Anchored Environment Generator” generates a physically constrained virtual environment that is molded and anchored to a real-world environment around a user (or multiple users); ¶26: real-world objects may also be visible as augmented video pass-through captured from sensing devices) comprising:
A hardware computer processor (Shapira, Fig. 6 and ¶¶212-213 discloses computing device 600 including processors for implementing Anchored Environment Generator); and
A non-transitory computer readable medium having software instructions stored thereon, the software instructions executable by the hardware computer processor to cause the AR system to perform operations (Shapira, Fig. 6 and ¶¶219-222 discusses CRM having software stored thereon embodying the Anchored Environment Generator) comprising:
Render a virtual environment to a user of the AR system (Shapira, ¶23: render physically constrained virtual environment to user via a display device; Also see ¶50);
Receive a selection of an item by the user (Shapira, ¶47: user interface module enables user or designer to define or select themes and virtual objects);
Automatically place the selected item at a location in the virtual environment of the user based at least in part on: affordances of the selected item; and affordances of objects associated with inanimate physical objects in the virtual environment of the user (Shapira, ¶24: dynamically modify the rendered content of the TVR to correspond to real-world environmental changes and/or user motion, position, and actions while interacting with the tactile virtual reality; ¶¶28-29: mapping of virtual objects to extents of real-world objects, using “affordances” where functional characteristics of underlying real-world objects may be incorporated into virtual objects for use by the user, even where those real-world objects are not visible to the user, including with inanimate objects such as a cell phone, seating surface, as inanimate physical objects tracked in the virtual environment, or alternatively things like a virtual sword or ornate throne as inanimate virtual objects in the virtual environment; ¶34: automated scene scanning and processing; ¶¶35-41 discloses the optimization of constraints of virtual objects in view of the contextual and semantic understanding of the real-world environment around the user, including geometric constraints relating to the real world environment and virtual objects, including semantic constraints relating to virtual object functionality and corresponding contextual relationships between real-world environment and those virtual objects, along with changes based on real-world environment or virtual content; also ¶¶88-89: objective constraints include virtual holographic displays on virtual tables or desks; also note ¶105 further discloses constraints between virtual objects).
Wherein automatically placing the selected item at the location in the virtual environment of the user includes applying virtual force to the selected item based on the affordances of the selected item and the affordances of the objects in the virtual environment of the user (Shapira, ¶3: molding and anchoring of a tactile virtual reality so that at least a portion of the TVR matches the tactile truth for one or more surfaces and objects within the real-world environment – discussed further in ¶32; Also ¶29 states: “virtual remodels of existing real-world spaces, walls, surfaces, buildings, outdoor spaces and landscaping, etc., can be rendered in the TVR having different colors, textures, patterns, combinations, scales and placements of virtual objects, etc. Note that the term “affordances” may be used herein to refer to such concepts. In other words, functional characteristics of underlying real-world objects may be incorporated into virtual objects for use by the user, even where those real-world objects are not visible to the user.”; ¶32 discusses anchoring virtual objects to match tactile truth of surfaces and objects in real-world environment; ¶¶48 and 50: molding module procedurally molds a physically constrained virtual environment to be consistent with the characteristics and constraints of the environmental information of the real world, including physically constraining virtual environment anchored to real-world environment; ¶68: real-time updates as user moves and interacts with TVR for dynamic and immersive virtual experience; ¶84: real surface can have semantic constraints and affordances for anchored virtual objects; ¶87-89 disclose that physical rest locations can be objective constraints, such as virtual displays on virtual tables or desks; ¶¶104-105 discloses use of pairwise constraints for two or more virtual objects; ¶138: pairwise constraints to avoid collisions with other virtual objects, or ensure certain distances between objects)
Wherein the virtual force includes an orientation (Shapira, ¶94 and ¶134 virtual objects can be transformed via translation, scaling, rotation; ¶137 further discloses constraints can be applied to objects, such as limiting rotations, such as certain directions etc.)
Wherein, upon determining that the placing of the selected item at the location is suitable, the automatically placing the selected item at the location in the virtual environment of the user further includes: calculating a distance between an initial location of the selected item from the virtual menu of items and the location, determining whether the distance between distance is less than a first threshold distance, and rendering, when the distance is less than the first threshold distance, a orientation of the selected item to align one or more objects at the location by automatically orientating the selected item to cause a surface normal of the selected item to face the AR system, the surface normal being defined by the affordances of the selected item; automatically orientating the selected item to cause a surface normal of the selected item to be perpendicular to a surface of the location; or automatically orientating the selected item to cause a surface normal of the selected item to be parallel to a normal of the location (Shapira, ¶72: geometric constraints ensures an object matches x,y,z coordinates; ¶85: the semantic constraints and affordances associated with the real table may be that virtual objects anchored to the location of the real table are intended to be placed on a corresponding virtual surface anchored to the location of that real table within the corresponding TVR, and that the user may interact with the table (e.g., set items on the table sit on the table etc.; ¶¶151-158 discloses the constructing of the TVR including reconstructing a scene as a 3D mesh or model representing the real-world environment, and models of virtual objects with transformations into the virtual space afforded by the corresponding real-word objects; ¶¶159-162 discloses constructing the TVR from the scene and room layout inputs by constructing a water-tight (i.e., enclosed space) representation of the environment, where the representation may be a set of planes, each labeled as floor, wall, ceiling, etc., all of which are guaranteed to be adjacent to another plane on all edges (therefore enclosed), and the scene layout interference algorithm includes ensuring that walls “stick” to observed walls (for tactile truth) – i.e. lining up of virtual planes inherently results in “automatically orientating the selected item to cause a surface normal of the selected item to be parallel to a normal of the location” as a mathematical principle of aligning two planes to correspond, where ¶168:
the Anchored Environment Generator combines real-world environments with virtual reality in a way that maps a virtual reality environment to objects, geometry, people, etc., within a real-world space. These features provide tactile truth to the resulting physically constrained virtual environment being presented to the user by the Anchored Environment Generator without actually displaying the underlying real-world environment, thereby enabling virtually limitless usage scenarios.
¶170: As another example, ¶170 discloses Real paper scrolls can then be placed in different real-world locations (and rendered as virtual scrolls in corresponding locations in the TVR). Users can then pick up and unroll those virtual scrolls (via the corresponding real scrolls), and the virtual content is rendered on those scrolls – i.e. the system requires rendering the virtual elements as user picks up the item in a way that “automatically orientating the selected item to cause a surface normal of the selected item to be parallel to a normal of the location”).
Shapira does not explicitly teach the selection of menu of items using a focus indicator as claimed (rather than some other type of selection).
Wheeler discloses:
Present a virtual menu of items to the user of the AR system (Wheeler, ¶¶16-17: augmented reality device including HMD displaying user interface, including interactive elements taking the form of a menu – See Fig. 7B);
Generate a focus indicator comprising a graphical representation of a cursor, virtual cone, or virtual ray; display the focus indicator at a location in the virtual environment of the user associated with a current pose of the user (Wheeler, ¶71: predetermined movements may take form of the user’s head; Figs. 7B and 7D and ¶75: user interface may include a cursor 210 which may be controlled by a user of the wearable computing device through one or more predetermined movements)
Receive a selection of an item by the user from the virtual menu of items based on the focus indicator by the user (Wheeler, Figs. 7B and 7D and ¶75: wearable computing device configured to receive selection data from user corresponding to a selection of a menu object 206 from menu 204, where user interface may include a cursor 210 which may be navigated around the view region to select menu objects)
Both Shapira and Wheeler are directed to augmented reality systems for user interaction of objects. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, by incorporating user position information for determining selection of an object using a menu of user interface elements as provided by Wheeler, using known electronic interfacing and programming techniques. The modification results in an improved user interface for placing virtual objects by providing an easier to use menu for better organizing and displaying items for user interaction using a more intuitive input technique for easier operation.
Wheeler further discloses:
Wherein the virtual force replicates at least one of a spring force, a gravitational force, an adhesive force, or an electromagnetic force. (Wheeler, ¶77: the one or more menu objects 206, and/or other objects in the user-interface 200 may function as "gravity wells," such that when the cursor 210 is within a predetermined distance of the object, the cursor is pulled toward the object by "gravity.")
Shapira and Wheeler are directed to augmented reality systems for user interaction with objects. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, by incorporating a menu for selection of the object as provided by Wheeler, and further including the simulation of a force for placing a virtual object as further provided by Wheeler, using known electronic interfacing and programming techniques. The modification provides an improved augmented reality placement of objects by allowing for an attraction force to be replicated such that virtual objects are more easily placed relative to other objects for easier user control.
Shapira in view of Wheeler does not explicitly teach that an orientation speed for the movement of a virtual object is at least partially dependent on a user preference associated with an orientation speed. This, however, is a known technique for manipulating objects within three dimensional space.
Yamaguchi discloses:
Wherein the virtual force is at least partially dependent on a user preference associated with an orientation speed (Yamaguchi, Abstract discloses user defined rotation; ¶114: rotation speed, r is based on alpha L, where user can freely set the value of coefficient alpha; ¶117 further discloses user input defining the speed of rotation for object rotation)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, by incorporating a menu for selection of the object as provided by Wheeler, and the simulation of a force for placing a virtual object as further provided by Wheeler, by further allowing a user to set an orientation speed of the object’s movement as provided by Yamaguchi, using known electronic interfacing and programming techniques. The modification merely applies a known technique for allowing user control over virtual object movement to a system for moving a virtual object on a user interface, yielding predictable results of merely applying a known user control over virtual object movement to a system and technique for manipulating virtual objects within a user interface that controls the visualization of a virtual object placement in a user’s view. The modification provides an improved virtual manipulation of a virtual object by allowing a user greater control over the dynamic movement to allow a more pleasing view of the object movement as set by user preference.
Rolleston discloses:
applying virtual force to the selected item based on the affordances of the selected item and the affordances of the objects in the virtual environment of the user, wherein the applying the virtual force to the selected item based on the affordances of the selected item and the affordances of the object in the virtual environment of the user includes applying a repulsive force when the affordances of the selected item and the affordances of the object in the virtual environment of the user indicates that one or more of the selected item and the objects in the virtual environment of the user at the location indicates that the placing of the selected item at the location is unsuitable (Rolleston, ¶10: virtual objects can be displaced so that objects do not pass through one another when a real world marker comes into close spatial proximity and the corresponding virtual objects begin to collide; ¶45: The spring connector 470 displaces the virtual objects 460 so that the objects 460 do not pass through one another when the real world markers 450 come into a dose spatial proximity, and the corresponding virtual objects 460 begin to collide; ¶48 further discloses that a virtual object tracks a marker when no real world collisions occur between markers, but virtual objects are displaced so that the virtual objects do not pass through one another when the real world markers come into a dose spatial proximity – Examiner notes that this teaches virtual objects repel one another so that they do not overlap, despite indication by a user moving a marker to try to place a virtual object at a particular location, where the repelling indicates the location is unsuitable, which read on by the claim limitation, further explained by Rolleston in Figs. 11-13 and ¶49, where objects that collide result in shift apart, or repulsion, as in fig. 13)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, by incorporating a menu for selection of the object as provided by Wheeler, and the simulation of a force for placing a virtual object as further provided by Wheeler, and allowing a user to set an orientation speed of the object’s movement as provided by Yamaguchi, by further incorporating the use of repulsion for ensuring virtual objects are not misplaced based on other objects as taught by Rolleston, using known electronic interfacing and programming techniques. The modification results in an improved interactive augmented reality experience with a plurality of objects to ensure more realistic interaction of objects and avoiding clunky overlaps, collisions or other misplacements of virtual objects that might otherwise produce unrealistic or distorted visual effects.
Ooi discloses:
Wherein, upon determining that the placing of the selected item at the location is suitable, the automatically placing the selected item at the location in the virtual environment of the user further includes: calculating a distance between an initial location of the selected item from the virtual menu of items and the location, determining whether the distance between distance is less than a first threshold distance, and rendering, when the distance is less than the first threshold distance, a orientation of the selected item to align one or more objects at the location by automatically orientating the selected item to cause a surface normal of the selected item to face the AR system, the surface normal being defined by the affordances of the selected item; automatically orientating the selected item to cause a surface normal of the selected item to be perpendicular to a surface of the location; or automatically orientating the selected item to cause a surface normal of the selected item to be parallel to a normal of the location (Ooi, ¶10: normal line of at least one face of virtual object becomes parallel with normal line of first face; Fig. 2 and ¶¶66-67: inserting virtual object into real space; ¶121: it is presumed that an instance of a virtual object is arranged such that it has the same tilt as a contacted face of a real object in the input image (that is, the two normal lines are parallel with each other))
Shapira, Wheeler, Rolleston and Ooi are directed to augmented reality systems for user interaction with objects. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, incorporating a menu for selection of the object and including the simulation of a force for placing a virtual object as further provided by Wheeler, further allowing a user to set an orientation speed of the object’s movement as provided by Yamaguchi and incorporating the use of repulsion for ensuring virtual objects are not misplaced based on other objects as taught by Rolleston, by further incorporating the technique for aligning virtual and physical objects as provided by Ooi, using known electronic interfacing and programming techniques. The modification results in an improved user interface for an augmented reality system by providing better visual alignment by coordinating the direction of objects to better line up for an improved visual presentation of virtual objects within a real-world scene, thus producing a more realistic and pleasing image.
Regarding claim 16, the operations of claim 1 perform the method of claim 16 and as such, claim 16 is rejected based on the same rationale as claim 1 set forth above.
Regarding claim 3, Shapira further discloses:
Wherein the software instructions are further configured to determine the location based on at least one of: an actuation of a user input device or a pose of a user (Shapira, ¶4: TVR replaces view of real-world environment around user such that virtual objects in TVR a viewed in head mounted display; ¶28: virtual object tracks location of users movement of object – i.e. sword follows cell phone movement)
Regarding claim 7, Shapira further discloses:
Wherein the location is a physical object with an at least partially planer surface within the environment of the user (Shapira, ¶28: virtual sword molded and anchored to location of users real cell phone or other real object in real-world environment; ¶29 discusses use of other real-world environment objects as placement for virtual objects, including armchair, walls, surfaces, etc. – i.e. at least partially planar; ¶87-89 disclose that physical rest locations can be objective constraints, such as virtual displays on virtual tables or desks)
Regarding claim 8, Shapira further discloses:
Wherein the physical object comprises a wall or a table (Shapira, ¶29 discusses use of other real-world environment objects as placement for virtual objects, including armchair, walls, surfaces, etc.; ¶¶64-65 further discuss constraints including walls; ¶72: virtual crate on real-world table)
Rolleston discloses:
Wherein the applying the repulsive force occurs when the affordances of the selected item and the affordances of the object in the virtual environment of the user indicate that the placing of the selected item at the location is unsuitable due to the object in the virtual environment of the user already occupying the location (Rolleston, ¶10: virtual objects can be displaced so that objects do not pass through one another when a real world marker comes into close spatial proximity and the corresponding virtual objects begin to collide; ¶45: The spring connector 470 displaces the virtual objects 460 so that the objects 460 do not pass through one another when the real world markers 450 come into a dose spatial proximity, and the corresponding virtual objects 460 begin to collide; ¶48 further discloses that a virtual object tracks a marker when no real world collisions occur between markers, but virtual objects are displaced so that the virtual objects do not pass through one another when the real world markers come into a dose spatial proximity – Examiner notes that this teaches virtual objects repel one another so that they do not overlap, despite indication by a user moving a marker to try to place a virtual object at a particular location, where the repelling indicates the location is unsuitable, which read on by the claim limitation, further explained by Rolleston in Figs. 11-13 and ¶49, where objects that collide result in shift apart, or repulsion, as in fig. 13)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, by incorporating a menu for selection of the object as provided by Wheeler, and the simulation of a force for placing a virtual object as further provided by Wheeler, and allowing a user to set an orientation speed of the object’s movement as provided by Yamaguchi, by further incorporating the use of repulsion for ensuring virtual objects are not misplaced based on other objects as taught by Rolleston, using known electronic interfacing and programming techniques. The modification results in an improved interactive augmented reality experience with a plurality of objects to ensure more realistic interaction of objects and avoiding clunky overlaps, collisions or other misplacements of virtual objects that might otherwise produce unrealistic or distorted visual effects.
Regarding claim 9, Shapira further discloses:
Wherein the location is a virtual object (Shapira, ¶85: semantic constraints and affordances associated with the real table may be that virtual objects anchored to the location of the real table are intended to be placed on a corresponding virtual surface anchored to the location of that real table within the corresponding TVR, and that the user may interact with the table (e.g., set items on the table sit on the table etc.) – i.e. virtual objects placed on virtual surface; ¶¶88-89: objective constraints include virtual holographic displays on virtual tables or desks)
Regarding claim 11, Shapira further discloses:
Wherein the affordances of the selected item are associated with one or more of the following: a function, an orientation, a type, a location, a shape, a size, or an environment of the selected item (Shapira, ¶¶28-29: mapping of virtual objects to extents of real-world objects, using “affordances” where functional characteristics of underlying real-world objects may be incorporated into virtual objects for use by the user, even where those real-world objects are not visible to the user, including molded to and anchored to location of real object, e.g. phone; ¶¶35-41 discloses the optimization of constraints of virtual objects in view of the contextual and semantic understanding of the real-world environment around the user, including geometric constraints relating to the real world environment and virtual objects, including semantic constraints relating to virtual object functionality and corresponding contextual relationships between real-world environment and those virtual objects, along with changes based on real-world environment or virtual content; ¶¶99-105 further disclose affordances based on the virtual item type, e.g. virtual trashcans belong on floor, or distance between virtual objects)
Regarding claim 20, the operations of claim 11 perform the method of claim 20 and as such, claim 20 is rejected based on the same rationale as claim 11 set forth above.
Regarding claim 12, Shapira further discloses:
Wherein the affordances of the objects in the virtual environment are associated with one or more of the following: a function, an orientation, a type, a location, a shape, a size, or an environment of the selected item (Shapira, ¶¶28-29: mapping of virtual objects to extents of real-world objects, using “affordances” where functional characteristics of underlying real-world objects may be incorporated into virtual objects for use by the user, even where those real-world objects are not visible to the user, including molded to and anchored to location of real object, e.g. phone; ¶¶35-41 discloses the optimization of constraints of virtual objects in view of the contextual and semantic understanding of the real-world environment around the user, including geometric constraints relating to the real world environment and virtual objects, including semantic constraints relating to virtual object functionality and corresponding contextual relationships between real-world environment and those virtual objects, along with changes based on real-world environment or virtual content).
Regarding claim 13, Shapira further discloses:
Wherein one or more physical attributes is assigned to the selected item, and the placement of the selected item is determined by simulating interactions of the selected item, the location, and the virtual environment based on the physical attributes of the selected item (Shapira, ¶24: dynamically modify the rendered content of the TVR to correspond to real-world environmental changes and/or user motion, position, and actions while interacting with the tactile virtual reality; ¶¶28-29: mapping of virtual objects to extents of real-world objects, using “affordances” where functional characteristics of underlying real-world objects may be incorporated into virtual objects for use by the user, such as virtual object tracks location of users movement of object – i.e. sword follows cell phone movement; ¶¶35-41 discloses the optimization of constraints of virtual objects in view of the contextual and semantic understanding of the real-world environment around the user, including geometric constraints relating to the real world environment and virtual objects, including semantic constraints relating to virtual object functionality and corresponding contextual relationships between real-world environment and those virtual objects, along with changes based on real-world environment or virtual content; also ¶¶88-89: objective constraints include virtual holographic displays on virtual tables or desks; also note ¶105 further discloses constraints between virtual objects)
Regarding claim 14, Shapira further discloses:
Wherein the one or more physical attributes assigned to the selected item (Shapira, ¶24: dynamically modify the rendered content of the TVR to correspond to real-world environmental changes and/or user motion, position, and actions while interacting with the tactile virtual reality; ¶¶28-29: mapping of virtual objects to extents of real-world objects, using “affordances” where functional characteristics of underlying real-world objects may be incorporated into virtual objects for use by the user, such as virtual object tracks location of users movement of object – i.e. sword follows cell phone movement; ¶¶35-41 discloses the optimization of constraints of virtual objects in view of the contextual and semantic understanding of the real-world environment around the user, including geometric constraints relating to the real world environment and virtual objects, including semantic constraints relating to virtual object functionality and corresponding contextual relationships between real-world environment and those virtual objects, along with changes based on real-world environment or virtual content; Shapira, ¶47: user interface module enables user or designer to define or select themes and virtual objects; also ¶¶88-89: objective constraints include virtual holographic displays on virtual tables or desks; also note ¶105 further discloses constraints between virtual objects)
Regarding claim 15, Shapira further discloses:
Wherein the software instructions are further configured to automatically orient the selected item to align one or more objects at the location by automatically orientating the selected item (Shapira, ¶24: dynamically modify the rendered content of the TVR to correspond to real-world environmental changes and/or user motion, position, and actions while interacting with the tactile virtual reality; ¶¶28-29: mapping of virtual objects to extents of real-world objects, using “affordances” where functional characteristics of underlying real-world objects may be incorporated into virtual objects for use by the user, even where those real-world objects are not visible to the user; ¶34: automated scene scanning and processing; ¶¶35-41 discloses the optimization of constraints of virtual objects in view of the contextual and semantic understanding of the real-world environment around the user, including geometric constraints relating to the real world environment and virtual objects, including semantic constraints relating to virtual object functionality and corresponding contextual relationships between real-world environment and those virtual objects, along with changes based on real-world environment or virtual content; also ¶¶88-89: objective constraints include virtual holographic displays on virtual tables or desks)
Shapira modified by Wheeler, Rolleston and Ooi further discloses:
Wherein the software instructions are further configured to automatically orient the selected item to align one or more objects at the location by automatically orientating the selected item to cause a surface normal of the selected item to face the AR system, the surface normal being defined by the affordances of the selected item; automatically orienting the selected item to cause a surface normal of the selected item to be perpendicular to a surface of the location; or automatically orienting the selected item to cause a surface normal of the selected item to be parallel to a normal of the location. (Ooi, ¶10: normal line of at least one face of virtual object becomes parallel with normal line of first face; Fig. 2 and ¶¶66-67: inserting virtual object into real space; ¶121: it is presumed that an instance of a virtual object is arranged such that it has the same tilt as a contacted face of a real object in the input image (that is, the two normal lines are parallel with each other))
Shapira, Wheeler, Rolleston and Ooi are directed to augmented reality systems for user interaction with objects. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, incorporating a menu for selection of the object and including the simulation of a force for placing a virtual object as further provided by Wheeler, further allowing a user to set an orientation speed of the object’s movement as provided by Yamaguchi and incorporating the use of repulsion for ensuring virtual objects are not misplaced based on other objects as taught by Rolleston, by further incorporating the technique for aligning virtual and physical objects as provided by Ooi, using known electronic interfacing and programming techniques. The modification results in an improved user interface for an augmented reality system by providing better visual alignment by coordinating the direction of objects to better line up for an improved visual presentation of virtual objects within a real-world scene, thus producing a more realistic and pleasing image.
Regarding claim 21, Shapira modified by Wheeler discloses:
Wherein automatically placing the selected item at the location in the virtual environment of the user further includes applying a virtual force to the selected item further based at least in part on affordances of the virtual environment, wherein the virtual environment includes at least one of gravity or air resistance (Wheeler, ¶77: the one or more menu objects 206, and/or other objects in the user-interface 200 may function as "gravity wells," such that when the cursor 210 is within a predetermined distance of the object, the cursor is pulled toward the object by "gravity")
Shapira and Wheeler are directed to augmented reality systems for user interaction with objects. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, by incorporating a menu for selection of the object as provided by Wheeler, and further including the simulation of a force for placing a virtual object as further provided by Wheeler, using known electronic interfacing and programming techniques. The modification provides an improved augmented reality placement of objects by allowing for an attraction force to be replicated such that virtual objects are more easily placed relative to other objects for easier user control.
Regarding claim 22, Shapira modified by Wheeler further discloses:
Wherein the virtual force on the selected item based on the affordances of the selected item defines at least one of an amount, speed, or acceleration of movement of the selected item to the location (Wheeler, ¶77: the one or more menu objects 206, and/or other objects in the user-interface 200 may function as "gravity wells," such that when the cursor 210 is within a predetermined distance of the object, the cursor is pulled toward the object by "gravity")
Shapira and Wheeler are directed to augmented reality systems for user interaction with objects. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, by incorporating a menu for selection of the object as provided by Wheeler, and further including the simulation of a force for placing a virtual object as further provided by Wheeler, using known electronic interfacing and programming techniques. The modification provides an improved augmented reality placement of objects by allowing for an attraction force to be replicated such that virtual objects are more easily placed relative to other objects for easier user control.
Claim(s) 2, 5, and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Shapira et al. (US 2016/0253842 A1) in view of
Wheeler et al. (US 2013/0246967 A1)
Yamaguchi et al. (US 2004/0164956 A1)
Rolleston et al. (US 2013/0155106 A1) and
Ooi (US 2012/0210255 A1) further modified by
West et al. (US 2017/0068323 A1)
Regarding claim 2, the limitations included from claim 1 are rejected based on the same rationale as claim 1 set forth above and incorporated herein. Further regarding claim 2, West discloses:
Wherein the software instructions are further configured to, upon selection, copy the selected item from the virtual menu of items into a virtual clipboard (West, ¶77 discloses that the object is dragged to virtual mini-board 702 – see Figs. 3A and 3B, which also adds to overall environment, but the mini board is reasonably interpreted as a virtual clipboard)
Shapira, Wheeler, Rolleston and West are directed to augmented reality systems for user interaction with objects.. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, incorporating a menu for selection of the object and including the simulation of a force for placing a virtual object as further provided by Wheeler, further allowing a user to set an orientation speed of the object’s movement as provided by Yamaguchi, and incorporating the use of repulsion for ensuring virtual objects are not misplaced based on other objects as taught by Rolleston, and incorporating the technique for aligning virtual and physical objects as provided by Ooi, by further incorporating the user of a mini board of the environment for placement of objects by the user as further provided by West, using known electronic interfacing and programming techniques. The modification results in an improved user interface for placing virtual objects by providing a smaller representation of the environment where copies of the virtual object are placed for easier understanding and control (see e.g. West. ¶27 discussing benefit of ability to see and interact with smaller view).
Regarding claim 5, the limitations included from claim 1 are rejected based on the same rationale as claim 1 set forth above and incorporated herein. Further regarding claim 5, Wheeler further discloses:
Identifying, using a pose sensor, a change in a pose of the user (Wheeler, ¶26: HMD tracking user head movement and/or eye movement using sensors)
Both Shapira and Wheeler are directed to augmented reality systems for user interaction of objects. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, by incorporating user position information for determining selection of an object using a menu of user interface elements as provided by Wheeler, using known electronic interfacing and programming techniques. The modification results in an improved user interface for placing virtual objects by providing an easier to use menu for better organizing and displaying items for user interaction using a more intuitive input technique for easier operation.
West further discloses:
Wherein the software instructions are further configured to determine the location based on receiving an indication from the user to place the selected item at the location that includes: identifying, using a pose sensor, a change in a pose of the user (West, ¶37: motion capture devices; ¶40: In some embodiments (e.g., in first-person perspective), the user 100 may see virtual representations of their hand(s) or arm(s) (not shown) as mirroring the real hands or arms of the user 100 (e.g., provided via motion tracking); ¶54: gesture of user touches and drags objects with hand or VR grasping tool, where user touches an object to move by dragging; Figs. 6A-6B and ¶60; ¶63: select by pointing hand; ¶77: drag and drop of virtual object; ¶108 further discloses touch inputs); identifying the location based at least partly on the pose of the user (West, ¶54: user touches location of touch on ground; ¶55: drag movement), and receiving a confirmation from the user to place the selected item at the location (West, ¶55: user unselects object, releasing or dropping at second location; Figs. 6A-6B and ¶60)
Shapira, Wheeler, Rolleston and West are directed to augmented reality systems for user interaction with objects. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, incorporating a menu for selection of the object and including the simulation of a force for placing a virtual object as further provided by Wheeler, further allowing a user to set an orientation speed of the object’s movement as provided by Yamaguchi, and incorporating the use of repulsion for ensuring virtual objects are not misplaced based on other objects as taught by Rolleston, by further incorporating the input technique of West, using known electronic interfacing and programming techniques. The modification results in an improved user interface for manipulating virtual objects by using a more intuitive and direct user control of objects.
Regarding claim 17, the limitations included from claim 16 are rejected based on the same rationale as claim 16 set forth above. Further regarding claim 17, Shapira further discloses:
Determining the location based on at least one of: an actuation of a user input device or a pose of a user (Shapira, ¶4: TVR replaces view of real-world environment around user such that virtual objects in TVR a viewed in head mounted display; ¶28: virtual object tracks location of users movement of object – i.e. sword follows cell phone movement)
Determining the selection of the item based on at least one of: an actuation of a user input device or pose of a user (Shapira, ¶47: user interface module enables user or designer to define or select themes and virtual objects; ¶79: manual user input used to identify particular objects; ¶¶214-215 discloses computing device having input devices for user input, the input devices such as touchscreens, pointing devices, keyboards, etc.)
Poulos further discloses:
Wheeler discloses:
Determining the location based on at least one of: an actuation of a user input device or pose of a user (Wheeler, ¶71: predetermined movements may take form of the user’s head; Figs. 7B and 7D and ¶75: user interface may include a cursor 210 which may be controlled by a user of the wearable computing device through one or more predetermined movements and wearable computing device configured to receive selection data from user corresponding to a selection of a menu object 206 from menu 204, where user interface may include a cursor 210 which may be navigated around the view region to select menu objects)
Both Shapira and Wheeler are directed to augmented reality systems for user interaction of objects. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, by incorporating user position information for determining selection of an object using a menu of user interface elements as provided by Wheeler, using known electronic interfacing and programming techniques. The modification results in an improved user interface for placing virtual objects by providing an easier to use menu for better organizing and displaying items for user interaction using a more intuitive input technique for easier operation.
West discloses:
Wherein the software instructions are further configured to, upon selection, copy the selected item from the virtual menu of items into a virtual clipboard (West, ¶77 discloses that the object is dragged to virtual mini-board 702 – see Figs. 3A and 3B, which also adds to overall environment, but the mini board is reasonably interpreted as a virtual clipboard)
Shapira, Wheeler and West are directed to augmented reality systems for user interaction with objects. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, incorporating a menu for selection of the object and including the simulation of a force for placing a virtual object as further provided by Wheeler, further allowing a user to set an orientation speed of the object’s movement as provided by Yamaguchi, and incorporating the use of repulsion for ensuring virtual objects are not misplaced based on other objects as taught by Rolleston, by further incorporating the input technique of West, using known electronic interfacing and programming techniques. The modification results in an improved user interface for manipulating virtual objects by using a more intuitive and direct user control of objects.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Shapira et al. (US 2016/0253842 A1) in view of
Wheeler et al. (US 2013/0246967 A1)
Yamaguchi et al. (US 2004/0164956 A1),
Rolleston et al. (US 2013/0155106 A1)
Ooi (US 2012/0210255 A1) and
West et al. (US 2017/0068323 A1) further modified by
Ramsby et al. (US 2015/0268821 A1)
Regarding claim 6, the limitations included from claim 5 are rejected based on the same rationale as set forth above. Further regarding claim 6, Ramsby discloses:
Wherein the confirmation comprises at least one of a change in the pose of the user or a hand gesture on a user input device (Ramsby, ¶24: selection input can be performed by other means, such as the selection input may be made via a selection input button 118 located on the HMD device or another device such as remote controller, or alternatively using sudden changes in head movement)
Shapira, Wheeler, Rolleston, West and Ramsby are directed to augmented reality systems for user interaction with objects. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, incorporating a menu for selection of the object and including the simulation of a force for placing a virtual object as further provided by Wheeler, and incorporating the input technique of West, further allowing a user to set an orientation speed of the object’s movement as provided by Yamaguchi, and incorporating the use of repulsion for ensuring virtual objects are not misplaced based on other objects as taught by Rolleston, and incorporating the technique for aligning virtual and physical objects as provided by Ooi, by further utilizing the control of selection using a device such as further provided by Ramsby, using known electronic interfacing and programming techniques. The modification merely substitutes one known technique for user selection for another, yielding predictable results of utilizing a different known techniques for providing user control over selection. Moreover, the modification results in an improved interface for allowing a different type of user control for easier useability that better tailors the system to a preference for input.
Claim(s) 23 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Shapira et al. (US 2016/0253842 A1) in view of
Wheeler et al. (US 2013/0246967 A1),
Yamaguchi et al. (US 2004/0164956 A1),
Rolleston et al. (US 2013/0155106 A1) and
Ooi (US 2012/0210255 A1) further modified by
Fekete et al. (US 2014/0132633 A1).
Regarding claim 23, the limitations included from claim 1 are rejected based on the same rationale as claim 1 set forth above. Further regarding claim 23, Fekete discloses:
Wherein the items of the virtual menu include at least one of a virtual TV, a virtual audio player, or a virtual game (Fekete, Figs. 11-12 and ¶¶118-119 and 121 discloses tray region 120 including various furnishings for selection and placement, including television furnishing 108-11, which can be placed and controlled)
Shapira, Wheeler and Fekete are directed to computer systems for user interaction with virtual objects. Furthermore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality system for placing virtual objects based on user selection of an object as provided by Shapira, incorporating a menu for selection of the object and including the simulation of a force for placing a virtual object as further provided by Wheeler, further allowing a user to set an orientation speed of the object’s movement as provided by Yamaguchi, and incorporating the use of repulsion for ensuring virtual objects are not misplaced based on other objects as taught by Rolleston, and incorporating the technique for aligning virtual and physical objects as provided by Ooi, by further incorporating a menu of virtual objects as provided by Fekete, using known electronic interfacing and programming techniques. The modification results in an improved user interface for adding virtual objects by allowing for a greater diversity of controllable virtual objects that better tailors the environment to user preferences and allows for more interactivity.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM A BEUTEL whose telephone number is (571)272-3132. The examiner can normally be reached Monday-Friday 9:00 AM - 5:00 PM (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DANIEL HAJNIK can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM A BEUTEL/Primary Examiner, Art Unit 2616