DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Response to Amendment
Claim 22 has been amended to correct for the previous objection which is therefore withdrawn.
Claims 2, 6, 12 and 16 have been canceled and as such all rejections to the canceled claims are withdrawn as moot.
Response to Arguments
Applicant’s arguments, see applicant’s remarks, filed 5/12/2026, with respect to the rejection(s) of claim(s) 1, 3-5, 7-11, 13-15, 17-18, 20 and 22 under 35 USC 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Nuernberger et al. and Ekstrand et al.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1, 3-5, 7-10 and 22 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claim 1, the amended claim recites, in lines 8-10:
PNG
media_image1.png
108
684
media_image1.png
Greyscale
The claim is indefinite based on the amendment deleting “space”, as there is no meaning behind “analyze a real” and further, the claim refers back to a limitation for which there is no antecedent bases, namely “the real space.” As such, the claim is indefinite. For purposes of examination, the claim is interpreted as if “space” was not deleted in line 8 (i.e. “real space”).
Claims 3-5, 7-10 and 22 depend from claim 1 and are therefore indefinite for the same reasons.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-4, 10, 11, 13-14 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Burns et al. (US 2020/0082632 A1) in view of
Nuernberger et al. (US 2017/0287218 A1) and in further view of
Ekstrand et al. (US 2015/0241994 A1).
Regarding claim 1, Burns discloses:
A wearable electronic device (Burns, ¶26: HMD; ¶33: device 10 is head-mounted display – Fig. 2) comprising:
a display (Burns, ¶26: pass through or translucent display);
a camera (Burns, ¶33: HMD using cameras; ¶64: camera included in HMD); and
memory storing instructions; (Burns, ¶60: memory including programs)
at least one processor (Burns, Fig. 6 and ¶56: device 10 includes processing units 142);
wherein the instructions, when executed by the at least one processor individually or collectively, cause the wearable device to (Burns, ¶60: program modules; ¶63: processor performing operations by executing code stored in memory):
based on an image obtained through the camera, (Burns, ¶64: obtain image data representing physical environment using image sensor/captured by camera included in HMD) analyze a real [space], (Burns, ¶67: detects an attribute in the physical environment depicted in the view based on the location of the virtual element in the view)
based on a result of the analyzed real space, obtain a plurality of areas where a virtual object is displayable, (Burns, ¶67: identify walls, ceilings, floors, tables, ramps, planar surfaces, curved surfaces, round surfaces, textured surfaces, surfaces of particular colors or textures, etc. In some implementations, detecting the attribute includes identifying a classification of a portion of the physical environment depicted in the view using a classifier or identifying an amount of the portion upon which the virtual element can be positioned)
based on receiving a first user input for displaying the virtual object, (Burns, ¶66: the change may include placement of the virtual element at a location in the view relative to the physical environment depicted in the view. Moreover, the change may be received by the device via input positioning the virtual element at the location.) identify a first area among a plurality of areas (Burns, ¶35: “The modality of a virtual element in the content is based on the location of the virtual element relative to the content. In some implementations, the modality of a virtual element is determined when the virtual element is first inserted into the content, e.g., based on the initial position of the virtual element”)
identify a shape of the first area (Burns, ¶67: detecting the attribute includes detecting a surface, an adjacent surface or element, or a type of surface upon which the virtual element is positioned in the physical environment depicted in the view, e.g., determining whether the virtual element is positioned in free space or on a floor, table, wall, or ceiling . . . a machine-learning model is trained to identify walls, ceilings, floors, tables, ramps, planar surfaces, curved surfaces, round surfaces, textured surfaces, surfaces of particular colors or textures, etc.);
identify a first shape of the virtual object corresponding to the shape of the first area (Burns, ¶68: method 700 selects a modality (e.g., an appearance, function, or interactivity) of the virtual element based on the attribute – note ¶68 discussing attributes of surface;
¶42:
FIG. 1 illustrates how the modality of the virtual element depends on its location relative to one or more attributes (e.g., being on a surface, vertical surface, a horizontal surface, a wall, a floor, a ceiling, a table, in mid-air, etc.) of the CGR environment 15. In this example, the modality of the virtual element depends on whether the virtual element is positioned on end table 25, on wall 30, on floor 35, or in open 3D space 40.
¶43:
The modalities, e.g., appearances, functions, and interactive features, of the virtual element can be configured by the virtual element creator, for example, who may create different modality state definitions for each of multiple positional states (e.g., on horizontal surface, on vertical surface, in mid-air, etc.) associated with multiple CGR content attributes (e.g., surfaces, horizontal surfaces, vertical surfaces, walls, floors, tables, ceilings, etc.).
Figs. 1-4 and ¶48:
As with the examples of FIGS. 1 and 2, the CGR environment 15 depicted in the examples of FIGS. 3 and 4, includes a virtual element that has different modalities when placed in different positions relative to attributes of the CGR content. The modalities, e.g., appearances, functions, and interactive features, of the virtual element can be configured by the virtual element creator, for example, who may create different modality state definitions for each of multiple positional states (e.g., on horizontal surface, on vertical surface, in mid-air, etc.) associated with multiple CGR content attributes (e.g., surfaces, horizontal surfaces, vertical surfaces, walls, floors, tables, ceilings, etc.).
),
identify at least one function related to the virtual object and corresponding to the shape of the first area (Burns, ¶68: method 700 selects a modality (e.g., an appearance, function, or interactivity) of the virtual element based on the attribute – note ¶68 discussing attributes of surface; ¶¶42-43 and 48), and
display the virtual object with the first shape in the first area (Burns, ¶69: the method 700 updates the view on the display of the device such that the view includes the virtual element according to the selected modality; Figs. 1-4 and ¶¶44-46, e.g. based on the virtual element being positioned on the wall 30 and thus associated with a vertical surface attribute, the virtual element is displayed in the vertical surface modality 45 (e.g., as a weather sign), and ¶50-52: virtual element may provide modality-specific functions and interactive features, such as face of clock etc.; For example, if a user positions the virtual element near a horizontal surface of table 75 in CGR environment 15, the virtual element may be displayed in a horizontal surface modality 90. In the horizontal surface modality 90, the virtual element has the appearance of a clock radio)
(Burns, ¶35: The modality of a virtual element in the content is based on the location of the virtual element relative to the content. In some implementations, the modality of a virtual element is determined when the virtual element is first inserted into the content, e.g., based on the initial position of the virtual element. In some implementations, the modality of a virtual element is determined and changed when the virtual element is moved within the content; ¶39: When a user of device 10 places or repositions a virtual element within a CGR environment 15 or changes the CGR environment 15 around the virtual element, the modality of the virtual element changes or adapts; Figs. 1-4, ¶¶44-46, ¶50-52 and ¶69;
and based on the second user input being completed, display the virtual object with the second shape in the second area (Burns, Figs. 1-4 and ¶¶44-46, e.g. based on the virtual element being positioned on the wall 30 and thus associated with a vertical surface attribute, the virtual element is displayed in the vertical surface modality 45 (e.g., as a weather sign); ¶66 discloses placement of a virtual element at a location relative to the physical environment by receiving input positioning the virtual element at the location, to change the positioning, which inherently displays at a location based on the user input being completed – i.e. user stops input, object stops moving)
Burns does not explicitly disclose the identifying a first area meeting a set condition among the plurality of areas, wherein the set condition comprises one of: a largest area among the plurality of areas, an area where the virtual object is disposed statistically most frequently or an area where there is a history for a user to have disposed the virtual object most frequently.
Nuernberger however discloses:
identify a first area meeting a set condition among the plurality of areas, wherein the set condition comprises one of: a largest area among the plurality of areas, an area where the virtual object is disposed statistically most frequently or an area where there is a history for a user to have disposed the virtual object most frequently. (Nuernberger, ¶32: a 3D model 72 of at least a portion of the physical environment 32 may be generated by HMD device 18 and utilized to display and manipulate virtual objects 30 within the physical environment; ¶33: candidate anchor features 74 may be extracted from the image data 26, where the candidate anchor features correspond to physical features 34 in the real world physical environment 32, where at least one correspondence between a virtual anchor feature of a virtual object 30 and a corresponding candidate anchor feature 74 may be identified; ¶64: the virtual object manipulation program 12 may identify at least one correspondence between a virtual anchor feature of the virtual object (cube 270) and a corresponding candidate anchor feature from a plurality of candidate anchor features in the room 210; ¶64: corresponding candidate anchor features that have been previously selected for use with a virtual anchor feature may be favored over candidate anchor features that have not been previously selected or have been selected less often, e.g., where the user 200 has placed virtual objects on the table top 236 twice as frequently as any other planar surface in the room 210, the corresponding candidate anchor feature of the table top 236 may be favored for selection over the other planar surfaces. )
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality user interface for relocating and changing virtual objects as provided by Burns, with the technique of identifying a preferred anchor location for selection by the system for placing a virtual object using placement history as provide by Nuernberger, using known electronic interfacing and programming techniques. The modification results in an improved placement of a virtual object in augmented reality for easier usability, and by better accounting for user context, allowing for easier placement of the object in an expected or preferred location.
The only limitation not explicitly taught is that the change in shape is performed while receiving a second user input to move the virtual object as claimed. However, changing shape based on proximity to a second location while being moved is a known technique.
Ekstrand discloses:
while receiving a second user input to move the virtual object with the first shape from the first area to a second area that is different from the first area, change the first shape to a second shape corresponding to a shape of the second area, based on the virtual object with the first shape approaching the second area; (Ekstrand, ¶35: determine that the cursor focus moves toward a target icon, and if a distance between the cursor focus and the target icon is greater than zero and is less than or equal to a first threshold, determine that the cursor focus selects the target icon; ¶38: to further improve user experience and a dynamic display effect on the screen, after the terminal device determines that the cursor focus moves toward the target icon, if the distance between the cursor focus and the target icon is less than or equal to a second threshold, a change of the cursor focus may be dynamically displayed on the screen, [such that] after the distance between the cursor focus and the target icon reaches a certain range, an effect of a change of the cursor focus may be dynamically displayed, [including] at least one of a size change of the cursor focus, a shape change of the cursor focus, and a color change of the cursor focus - see fig. 3)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality user interface for relocating and changing virtual objects as provided by Burns, with the technique of identifying a preferred anchor location for selection by the system for placing a virtual object using placement history as provide by Nuernberger, by further using a relative distance thresholding condition for changing the shape of a moved virtual object to a destination as provided by Ekstrand, using known electronic interfacing and programming techniques. The modification results in an improved graphical user interface by “improve user experience and a dynamic display effect on the screen” (Ekstrand, ¶35), and also providing improved guidance to a user during a move operation to more quickly show a changed element as it is moved so that a user can determine whether they want the change or not quicker and requiring less time and input effort.
Regarding claim 11, the device of claim 1 performs the method of claim 11 and as such claim 11 is rejected based on the same rationale as claim 1 set forth above.
Regarding claim 3, Burns further discloses:
wherein at least one or more icons included in the virtual object with the second shape or type of a function corresponding to the virtual object with the second shape is different from that of the virtual object with the first shape. (Burns, Figs. 1-4 and ¶43: The modalities, e.g., appearances, functions, and interactive features, of the virtual element can be configured by the virtual element creator, for example, who may create different modality state definitions for each of multiple positional states (e.g., on horizontal surface, on vertical surface, in mid-air, etc.) associated with multiple CGR content attributes (e.g., surfaces, horizontal surfaces, vertical surfaces, walls, floors, tables, ceilings, etc.).; ¶¶44-45 discusses various functionality and virtual element features displayed based on surface modality – see Fig. 1)
Regarding claim 13, the device of claim 3 performs the method of claim 13 and as such claim 13 is rejected based on the same rationale as claim 3 set forth above.
Regarding claim 4, Burns further discloses:
wherein an icon corresponding to a first function included in the virtual object with the first shape is a 2D icon, and an icon corresponding to the first function included in the virtual object with the second shape is a 3D icon. (Burns, Fig. 1 and ¶¶45-46: In contrast to the 2D appearance of the vertical surface modality 45, the open space modality 50 has a 3D appearance, e.g. the open space modality 50 provides a 3D representation of the current weather or predicted weather conditions in the user's current geographic location, e.g., displaying a floating sun, a rain cloud, a tornado, etc.; Note ¶¶41-42 further explains Fig. 1 modality based on location)
Regarding claim 14, the device of claim 4 performs the method of claim 14 and as such claim 14 is rejected based on the same rationale as claim 4 set forth above.
Regarding claim 10, Burns further discloses:
Wherein the first area has a shape corresponding to a bottom surface, a shape corresponding to a wall surface, or a shape corresponding to the virtual object being disposed mid-air and wherein the second area has a shape different from the first area (Burns, Fig. 1-4 and ¶¶44-46, shape shown based on wall surface or in air)
Regarding claim 20, the device of claim 10 performs the method of claim 20 and as such claim 20 is rejected based on the same rationale as claim 10 set forth above.
Claim(s) 5 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Burns et al. (US 2020/0082632 A1) in view of
Nuernberger et al. (US 2017/0287218 A1) and
Ekstrand et al. (US 2015/0241994 A1) in further view of
Berliner et al. (US 2022/0253093 A1).
Regarding claim 5, the limitations included from claim 1 are rejected based on the same rationale as claim 1 set forth above. Further regarding claim 5, Burns further discloses:
wherein the at least one processor is further configured to: based on receiving a third user input for adding the icon related to a second function receiving a fourth user input for adding the 2D icon related to the second function(Burns, ¶35: In some implementations, the modality of a virtual element is determined when the virtual element is first inserted into the content, e.g., based on the initial position of the virtual element; ¶44: the vertical surface modality 45 of the virtual elements displays a 2D image or sign representing the weather and the high and low predicted daily temperatures; ¶46: The virtual element in this example is positioned relative to a horizontal flat surface of depiction of a real-world end table 25. Based on this location, the virtual element is displayed in a horizontal surface modality 55, e.g., as a decorative snow globe in which the current or predicted weather is displayed in a 3D manner within the globe; i.e. icon either 2D or 3D based on inserted position, where user places virtual element; See ¶39: “When a user of device 10 places or repositions a virtual element within a CGR environment 15 or changes the CGR environment 15 around the virtual element, the modality of the virtual element changes or adapts” such that input for adding based on different locations are inputs; ¶50 also discloses user can customize modalities for how virtual element is displayed, user presented with modality-specific display options)
Burns does not explicitly disclose adding an icon to the objects based on user input as claimed. Adding icons related to functions to virtual objects based on user input, however, would have been known at the time of the effective filing date of the claimed invention.
Berliner discloses:
based on receiving a user input for adding the icon related to a function to the virtual object, add an icon related to the function to the virtual object of the shape, (Berliner, ¶550: controlling virtual display includes adding or deleting elements from the display; ¶746: docking at least one virtual object to the virtual display, includes adding the virtual object to the data-structure of virtual objects docked to the virtual display, or “docking a first virtual object to a second virtual object may include adding the first virtual object to a data-structure of virtual objects docked to the second virtual object (such as a list, a set, a database, and so forth)” Fig. 65 and ¶795: virtual objects docked to positions in a virtual plane, where virtual objects 6512 associated with first virtual plane 6510)
The combination of the references teaches the full limitation of the claim, wherein the at least one processor is further configured to: based on receiving a third user input for adding an icon related to a second function to the virtual object of the first shape, add a 2D icon related to the second function to the virtual object of the first shape, and based on receiving a fourth user input for adding the 2D icon related to the second function to the virtual object of the second shape, add a 3D icon related to the second function to the virtual object of the second shape.
Both Burns and Berliner are directed user interfaces for user manipulation of virtual objects within augmented reality systems. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality user interface for relocating and changing virtual objects as provided by Burns, using the technique of identifying a preferred anchor location for selection by the system for placing a virtual object using placement history as provide by Nuernberger, and using a relative distance thresholding condition for changing the shape of a moved virtual object to a destination as provided by Ekstrand, with the technique of docking virtual objects to other virtual objects as provided by Berliner, using known electronic interfacing and programming techniques. The modification results in an improved augmented reality user interface by allowing a user to associate functional objects together to build a tailored interface that better suits the user’s needs, for greater flexibility and usage of an augmented reality user interface.
Regarding claim 15, the device of claim 5 performs the method of claim 15 and as such claim 15 is rejected based on the same rationale as claim 5 set forth above.
Claim(s) 7 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Burns et al. (US 2020/0082632 A1) in view of
Nuernberger et al. (US 2017/0287218 A1) and
Ekstrand et al. (US 2015/0241994 A1) and in further view of
Schwarz et al. (US 2018/0286126 A1).
Regarding claim 7, the limitations included from claim 1 are rejected based on the same rationale as claim 1 set forth above and incorporated herein. Further regarding claim 7, Burns does not explicitly disclose based on approach of the virtual object of the first shape from the first area to around the second area through the second user input, display a virtual object indicating that a virtual object is fixable in the second area. This limitation is essentially claiming anchoring of an object upon the completion of a user input, which is well-known in the field of augmented reality.
Schwarz discloses:
wherein the at least one processor is further configured to: based on approach of the virtual object of the first shape from the first area to around the second area through the second user input, display a virtual object indicating that a virtual object is fixable in the second area (Schwarz, Figs. 9-10 and ¶68: With reference now to FIGS. 9 and 10, in some examples when the user interface element layout program 12 determines that one or more of the virtual object and the one or more UI elements are within the predetermined distance of a physical surface, the program may display one or more of (1) a visual indication with the one or more user interface elements and (2) a visual indication on the physical surface; ¶69: On making this determination, the user interface element layout program 12 may display a visual indication with the user interface elements 76, such as a highlighted border 604 around the elements. In other examples, any other visual indication may be provided, such as adding color to or changing a color of the UI elements 76. In this manner, visual feedback is provided to the user to alert the user that the UI elements 76 may be transitioned to the table top 236).
Both Burns and Schwarz are directed user interfaces for user manipulation of virtual objects within augmented reality systems. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality user interface for relocating and changing virtual objects as provided by Burns, using the technique of identifying a preferred anchor location for selection by the system for placing a virtual object using placement history as provide by Nuernberger, and using a relative distance thresholding condition for changing the shape of a moved virtual object to a destination as provided by Ekstrand, with the technique of providing indicators to inform user that virtual object can be placed on an object as provided by Schwarz, using known electronic interfacing and programming techniques. The modification results in an improved augmented reality user interface for relocating virtual objects within a physical space by providing better indicators to user as to functionality and to better assist a user with understanding where virtual objects can be placed for easier usability.
Regarding claim 17, the device of claim 7 performs the method of claim 17 and as such claim 17 is rejected based on the same rationale as claim 7 set forth above.
Claim(s) 8-9 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Burns et al. (US 2020/0082632 A1) in view of
Nuernberger et al. (US 2017/0287218 A1) and
Ekstrand et al. (US 2015/0241994 A1) and in further view of
Cazamias et al (US 2023/0343027 A1).
Regarding claim 8, the limitations included from claim 1 are rejected based on the same rationale as claim 1 set forth above and incorporated herein. Further regarding claim 8, Cazamias discloses:
wherein the at least one processor is further configured to: based on a virtual object related to a second application, that is different from a first application corresponding to the virtual object with the first shape, approaching the virtual object with the first shape in a state that the virtual object qith the first shape is displayed, automatically align the virtual object corresponding to the second application in an area around the virtual object with the first shape. (Cazamias, ¶24: the user 104 may use gestures to move the virtual object 110a from a first position to a second position, as indicated by the solid arrow in FIG. 1C, e.g. the displayed movement is based on the first gesture 112 and the displayed movement may follow a direction of the first gesture 112, where the electronic device 102 detects a movement of the virtual object 110a within a threshold distance of another virtual object; Fig. 1E and ¶26: display grouped objects along a line; ¶39: when the object placement determiner 330 determines that the first virtual object has moved within the threshold distance of the second virtual object, the object placement determiner 330 associates the first virtual object and the second virtual object, e.g., creates a group comprising the first virtual object and the second virtual object; ¶40: a display module 340 causes the display 302 to display virtual objects (e.g., the first virtual object and the second virtual object) at the object placement locations determined by the object placement determiner 330; ¶50: movement of grouped objects toward a point between the two objects – i.e. automatically align; ¶57 discusses drop zone)
Both Burns and Cazamias are directed user interfaces for user manipulation of virtual objects within augmented reality systems. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality user interface for relocating and changing virtual objects as provided by Burns, using the technique of identifying a preferred anchor location for selection by the system for placing a virtual object using placement history as provide by Nuernberger, and using a relative distance thresholding condition for changing the shape of a moved virtual object to a destination as provided by Ekstrand, with the technique of associating virtual objects together within the augmented interface for coordinated interaction and display as provide by Schwarz, using known electronic interfacing and programming techniques. The modification results in an improved user interface by allowing a user to more easily manipulate movement and placement of multiple objects in augmented reality for easier usability.
Regarding claim 9, Burns modified by Nuernberger, Ekstrand and Cazamias further discloses:
wherein the at least one processor is further configured to display a virtual object indicating that the virtual object with the first shape and the aligned virtual object corresponding to the second application are grouped. (Cazamias, ¶24: the user 104 may use gestures to move the virtual object 110a from a first position to a second position, as indicated by the solid arrow in FIG. 1C, e.g. the displayed movement is based on the first gesture 112 and the displayed movement may follow a direction of the first gesture 112, where the electronic device 102 detects a movement of the virtual object 110a within a threshold distance of another virtual object; Fig. 1E and ¶26: display grouped objects along a line; ¶39: when the object placement determiner 330 determines that the first virtual object has moved within the threshold distance of the second virtual object, the object placement determiner 330 associates the first virtual object and the second virtual object, e.g., creates a group comprising the first virtual object and the second virtual object; ¶40: a display module 340 causes the display 302 to display virtual objects (e.g., the first virtual object and the second virtual object) at the object placement locations determined by the object placement determiner 330; ¶50: movement of grouped objects toward a point between the two objects – i.e. automatically align; ¶57 discusses drop zone)
Both Burns and Cazamias are directed user interfaces for user manipulation of virtual objects within augmented reality systems. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality user interface for relocating and changing virtual objects as provided by Burns, using the technique of identifying a preferred anchor location for selection by the system for placing a virtual object using placement history as provide by Nuernberger, and using a relative distance thresholding condition for changing the shape of a moved virtual object to a destination as provided by Ekstrand, with the technique of associating virtual objects together within the augmented interface for coordinated interaction and display as provide by Schwarz, using known electronic interfacing and programming techniques. The modification results in an improved user interface by allowing a user to more easily manipulate movement and placement of multiple objects in augmented reality for easier usability.
Regarding claim 18, the device of claim 9 performs the method of claim 18 and as such claim 18 is rejected based on the same rationale as claim 9 set forth above.
Claim(s) 22 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Burns et al. (US 2020/0082632 A1) in view of
Nuernberger et al. (US 2017/0287218 A1) and
Ekstrand et al. (US 2015/0241994 A1) and in further view of
Khoe et al. (US 2013/0050263 A1).
Regarding claim 22, the limitations included from claim 1 are rejected based on the same rationale as claim 1 set forth above. Further regarding claim 22, Burns further discloses:
Wherein the instructions, when executed by the at least one processor individually or collectively, cause the wearable electronic device to: (Burns, ¶33: HMD using cameras; ¶64: camera included in HMD; ¶60: program modules; ¶63: processor performing operations by executing code stored in memory) identify at least one function related to the virtual object and corresponding to the shape of the first area, (Burns, Fig. 1 and ¶44: virtual element displayed in a vertical surface modality, where the virtual element displays a 2D image representing weather and high/low and predicted daily temperatures, where “functionality and interactivity of the virtual element may depend upon its modality”, e.g. the vertical surface modality of the virtual element may provide modality-specific functionality, such as a display of a Doppler radar map; This is as opposed to a horizontal flat surface modality– see ¶46; also note ¶48: The modalities, e.g., appearances, functions, and interactive features, of the virtual element can be configured by the virtual element creator, for example, who may create different modality state definitions for each of multiple positional states (e.g., on horizontal surface, on vertical surface, in mid-air, etc.) associated with multiple CGR content attributes (e.g., surfaces, horizontal surfaces, vertical surfaces, walls, floors, tables, ceilings, etc.); Also, shape is taught by ¶67:
At block 740, the method 700 detects an attribute in the physical environment depicted in the view based on the location of the virtual element in the view. In some implementations, detecting the attribute includes detecting a surface, an adjacent surface or element, or a type of surface upon which the virtual element is positioned in the physical environment depicted in the view, e.g., determining whether the virtual element is positioned in free space or on a floor, table, wall, or ceiling. In some implementations, detecting the attribute includes using a model, such as a neural network or other machine-learning model, to identify objects and other attributes of the physical environment. In some implementations, a machine-learning model is trained to identify walls, ceilings, floors, tables, ramps, planar surfaces, curved surfaces, round surfaces, textured surfaces, surfaces of particular colors or textures, etc. In some implementations, detecting the attribute includes identifying a classification of a portion of the physical environment depicted in the view using a classifier or identifying an amount of the portion upon which the virtual element can be positioned, e.g., detecting that the virtual element is positioned in mid-air (e.g., that the virtual element is not positioned on a surface).
)
Wherein the at least one function includes a first plurality of functions related to the virtual object, and the virtual object with the first shape includes a first plurality of icons corresponding to the first plurality of functions, (Burns, Fig. 2 and ¶47: wearable device for displaying objects; Fig. 3 and ¶¶49-52 discloses a display of a virtual element in different modalities based on positioning, placing along the wall 30 proximate to the floor, the modularity results in the virtual element changing from analog clock to analog and digital or visa-versa; ¶41: weather object in different modalities – Fig. 1 – which shows the position on the wall indicating both sunny, i.e. functionality of environment conditions, and temperature functionality, and when moved to other location, the shape changes to e.g. 50)
wherein the virtual object with the second shape includes a second plurality of icons (Burns, Fig. 2 and ¶47: wearable device for displaying objects; Fig. 3 and ¶¶49-52 discloses a display of a virtual element in different modalities based on positioning, placing along the wall 30 proximate to the floor, the modularity results in the virtual element changing from analog clock to analog and digital or visa-versa; ¶41: weather object in different modalities, including from 2D to 3D shapes)
The only limitation that is arguably not explicitly taught by Burns is wherein the virtual object with the second shape includes a second plurality of icons corresponding to a second plurality of functions at least one of which is different from the first plurality of functions, and wherein the second plurality of functions includes the first plurality of functions. In other words, an expansion of the displayed functionalities of a user interface element based on user input for a change of display of the user interface element. Examiner notes that Burns does teach changing the display of an analog clock element, which includes a plurality of icons representing different measurements of time, i.e. minutes vs. hours, which when moved is expanded to show the same analog clock icons for minutes and time, but also displaying the digital time function (see e.g. Fig. 3). For sake of clarity and compact prosecution, however, additional prior art is relied upon to show that expansion of user interface elements wherein the virtual object of the second shape includes a second plurality of icons corresponding to a second plurality of functions at least one of which is different from the first plurality of functions, and wherein the second plurality of functions includes the first plurality of functions would have been known and obvious to one of ordinary skill in the art.
Khoe discloses:
wherein the virtual object of the second shape includes a second plurality of icons corresponding to a second plurality of functions at least one of which is different from the first plurality of functions, and wherein the second plurality of functions includes the first plurality of functions (Khoe, Figs. 5S to 5T and ¶¶194-195: touch gesture detected on rotation user interface object of popup view 510, where in response to detecting the touch gesture on rotation user interface object 522, the portrait popup view 510 rotates so that it is displayed as landscape popup view, where by rotating second electronic device 100-2 from the portrait orientation to the landscape orientation, the displayed calculator application view changes from the simple calculator application view 512 (FIG. 5S) to the scientific calculator application view 526 (FIG. 5T) on second electronic device 100-2)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the augmented reality user interface for relocating and changing virtual objects as provided by Burns, with the technique of identifying a first area meeting a condition for a virtual object as provided by Mene, with the technique of expanding functionality of a user interface object based on a user input to change the viewing of the object as provided by Khoe, using known electronic interfacing and programming techniques. The modification merely applies a known technique of expanding user interface objects or widgets based on user input to an existing device that changes the viewing of a user interface object based on user input, to yield predictable results of providing a common expansion of functionality to graphical user interface elements based on user input. The expansion of user interface widgets to provide greater information and functionality to a user based on input is applicable to the base device that provides different functionality and views of graphical user interface elements or widgets in augmented reality basd on user input. The modification merely combines a known software modification to user interface elements within a graphical user interface design, and further allows for easier access to additional functionality that provides greater usability to a user without requiring complicated or time wasting input by the user themselves.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Faaborg et al. (US 2017/0256096 A1) is related to the present application as it is directed to determining locations to place a virtual object within a real-world environment based on characteristics of the virtual object and the real-world space in which the virtual object is inserted, including characteristics of the environment and user history (See e.g. Faaborg, ¶4 discloses computing device capturing feature information of an ambient environment, generating a 3D virtual model of the ambient environment to define a plurality of virtual drop targets, each associated with drop regions, and a request to place a virtual object in the 3D virtual model, where the computing device selects a virtual drop target, of the plurality of drop targets, for placing a virtual object in the 3D virtual model, based on the attributes of the virtual object and characteristic of the plurality of virtual drop targets, and displaying the sized virtual object at the selected virtual drop target in the displayed 3D virtual mode; Fig. 2 and ¶34 discloses position of the plurality of virtual display screens 220 in the horizontal arrangement, and/or the order of the vertical layering of the plurality of virtual display screens 220 may be based on, for example, historical usage that is collected, stored and updated by the system, and/or may be set by the user based on user preferences.)
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM A BEUTEL whose telephone number is (571)272-3132. The examiner can normally be reached Monday-Friday 9:00 AM - 5:00 PM (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DANIEL HAJNIK can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM A BEUTEL/Primary Examiner, Art Unit 2616
Prosecution Insights