Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities: the same reference number is used for multiple elements. For example “10” has been used to designate “ring”, “first ring”, “second ring”, and “new ring”.
Appropriate correction is required.
Drawings
The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “10” has been used to designate “ring”, “first ring”, “second ring”, and “new ring”. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claim 1 is objected to because of the following informalities: the claim introduces the limitation “the user” at line 5 using improper antecedent. Appropriate correction is required.
Claim 1 is objected to because of the following informalities: at line 16, the claim recites “the launcher selects the plurality of graphical elements”, which appears to be a typographical error, and should more appropriately state “the launcher selects one of the plurality of graphical elements”. Appropriate correction is required.
Claim 6 is objected to because of the following informalities: at line 1, the claim recites “A system according to claim 5” using improper antecedent basis since “system” was already introduced in claim 5, which depends on claim 1. Appropriate correction is required.
Claim 19 is objected to because of the following informalities: at line 9, the claim recites “a gaze” using improper antecedent basis since “gaze” was already introduced at line 2 of the claim. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The claims are generally narrative and indefinite, failing to conform with current U.S. practice. They appear to be a literal translation into English from a foreign document and are replete with grammatical and idiomatic errors.
For example:
Claim 1 recites “in particular”, which renders the claim indefinite. Appropriate correction is required.
By virtue of their dependency, claims 2-10 are also indefinite. Appropriate correction is required.
Claim 5 claim recites “optionally with image recognition and/or detection of electronic codes (barcode, QR codes, etc.);” rendering the claim indefinite.
By virtue of their dependency claims 6-7 are also indefinite. Appropriate correction is required.
Claim 8 recites “preferably securely to form a shared device cluster,” which renders the claim indefinite (MPEP 2173.05(d)). Appropriate correction is required.
Furthermore, claim 10 recites “an image capturing device” at line 3, “a display” in line 4, “at least one processor” at line 4, “an interface” at line 4, and “an at least partially virtual content item” at lines 5-6 of the claim, all which have been introduced in claim 1, from which claim 10 depends. There is insufficient/improper antecedent basis for these limitations in the claim. Appropriate correction is required.
Additionally, claim 10 recites the limitation “the plurality of visual interface devices (DI) capturing the real environment of the user in order to provide it to the interface”. It is not clear which “interface” is being referenced by “the interface” since each of the plurality of interface devices comprise an interface, and furthermore, because “the interface” was introduced in claim 1. As such, the claim is rendered indefinite.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 5-7, 9, 11-13, 15-17 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Han et al., US 2018/0081520 A1 (hereinafter “Han”) in view of Kashitani, US 2016/0012645 A1 (hereinafter “Kashitani”)
Regarding claim 1, Han discloses a human-machine interface system (FIGS. 1-4, electronic device 100 at [0033]-[0040] and [0054]-[0055]), comprising at least one visual interaction device (DI) (FIGS. 1-4, head mounted display 200 with electronic device 100 at [0066]-[0071] and [0073]-[0076] which has a display 131 that has a touch input and user input unit 120 at [0040]-[0046]), comprising an image capture device (FIGS. 1-3 and camera module 170 at [0034], [0057], [0059], [0108] and [0122]), a display (Abstract and FIGS. 1-4 display 131 at [0041]-[0046] and [0059] and [0064]), and at least one processor (FIGS. 1-5 and controller 180 at [0034], [0041], [0047], [0050] and [0059]-[0065]) running an interface (FIGS. 1-5 and visual output on display at [0041], GUI or user interface at [0032] and [0042]-[0043] and [0051] displayed by the display 131) for user interaction (FIGS. 1-5 and [0040]-[0041] and [0046]-[0052] and user input 120 with interaction and inputs described therein) with the at least one visual interaction device (DI) (FIGS. 1-4, head mounted display 200 with electronic device 100 at [0066]-[0071] and [0073]-[0076]),
the system presenting the user (FIG. 6 and user 611) with an at least partially virtual content item (FIGS. 6-8, [0103]-[0110] as presented in the display 131 to the user, content can be virtual or augmented reality at [0081], [0105] and [0121]), in particular, for augmented reality (FIGS. 5-6 and electronic device producing virtual reality or augmented reality at [0081], [0105] and [0121]),
the at least one visual interaction device (DI) (FIGS. 1-4, head mounted display 200 with electronic device 100) and tracks the user's gaze (FIGS. 5-9, and 11-13, and [0087] at operation 511 and [0097]-[0099], [0108] at operation 807, [0113]-[0114], and [0120]-[0122] operation 1103),
the interface being characterized in that it is configured to detect when the user's gaze is oriented toward the ground (FIG. 6 illustrating 615 with navigator at the feet around the user as disclosed at [0091]-[0093], generally FIG. 9 and [0112]-[0114]; and FIG. 8 at [0103]-[0104] operation 801 displaying the navigator according to the user’s gaze information) and, following this detection, to start a launcher (FIG. 8 at [0103]-[0104] operation 801 displaying the navigator according to the user’s gaze information) that is presented to the user via the at least one visual interaction device (DI) (FIG. 8, [0103]-[0111] describing presentation via the device 100 of navigator and content, FIG. 9, navigators 910, 920, 930, 940 at [0112]-[0114]) in a form of at least one navigation content item (FIG. 9, navigators 910, 920, 930, 940 at [0112]-[0114]) comprising a plurality of graphical elements (FIGS. 6, 9, 12, 13, e.g., object information 623 at [0093]-[0094], FIG. 9 and object information 935 and [0112]-[0114]) that the user is able to scroll through with their gaze (FIGS. 9-13, and [0103]-[0111] describing object selection at operation 807 therein; further describing updating the visible objects by determining the direction of the head and head-tracking at operation 803 and updating the visible objects on the display (e.g., scrolling through) accordingly) and that are representative of a function to which they correspond (FIGS. 11-13 and operations 1107-1111 and [0124]-[0129] and executing a function associated with the navigator and object at [0131]-[0133] with call, share, and view functions 1335),
the interface detecting movement of the user's gaze across the plurality of graphical elements (FIGS. 11-13, describing movement of user’s focus/gaze in operations 1103-1109 to determine whether user’s focus is maintained [0119]-[0126]), and, when the user's gaze has come to rest on one of the plurality of graphical elements for a given duration (FIGS. 11-13, operation 1109, determine whether user’s focus/gaze is maintained for a predetermined amount of time [0119]-[0126]), the launcher selects the plurality of graphical elements and triggers the function to which it corresponds (FIGS. 11-13, operation 1109-1111, determine whether user’s focus/gaze is maintained for a predetermined amount of time and executing the associated function with the object [0119]-[0126], further at least illustrating additional functions at FIG. 13 and additional information 1335 at [0131]-[0133]), a set of available functions (FIG. 13 and additional information/functions 1335 that are executable at [0131]-[0133]) comprising at least:
a) execution of an application available via the interface and executable on said the at least one visual interaction device (DI) (FIGS. 11-13; operation 1111 at [0126] for executing a function associated with the object information, such as playback of a moving image by the controller 180; additional information/functions 1335 that are executable at [0131]-[0133], the applications including call, share, view function, for example to the call function would be to call the person/contact, share would be send the location to the contact/counterpart);
b) opening of at least one content item or document stored in a non-transitory computer-readable storage device accessible via the interface (FIGS. 1-5 and 8-13 describing inputted/outputted data including files such as videos, images photos, audios, etc. at [0050] being stored in a memory 150, and retrieving the thumbnail images at least at [0110] and [0121]-[0126] to be displayed, operation 1111 of FIG. 11 displays/opens a list of executable functions associated with a contact at [0126]) for example gaze at 1331, causing 1333 to open at [0130]-[0133]); and
c) deployment of other graphical elements in the launcher, across which the user can move their gaze in order to trigger the same functions a), b) or c) when it comes to rest thereupon for the given duration (FIGS. 8-13 and [0111]-[0114] describing gaze on 935 creating a new navigator with different objects in the content different than the original; further at [0121]-[0133] the operations such as 1111 performable on these new objects accordingly).
However, although Han discloses augmented reality ([0081], [0105] and [0121]), Han does not explicitly disclose capturing a real environment of the user in order to provide it to the interface, which generates a mesh of the real environment.
In the same field of endeavor, Kashitani discloses a head mounted display ([0064]) with a display device D capable imaging an image captured by a camera ([0064]) by capturing a real environment of the user in order to provide it to the interface (FIGS. 1-3, [0064]-[0068] describing capturing the environment 1 by the camera mounted on the head and outputs the image to the display device D of the head mounted display HMD), which generates a mesh of the real environment (FIGS. 1-3, [0064]-[0068] the display, through an image processing device 100 display superimposition display data in a superimposed manner on actual space by optically compositing the superimposition display data within the visual field of the user).
Before the effective filing date, it would have been obvious to a person of ordinary skill in the art to modify the head mounted device for interaction by the user with displayed objects within the user’s visual field of Han to incorporate the real environment image capture and displaying as disclosed by Kashitani because the references are within the same field of endeavor, namely, virtual reality devices capable of user interaction with the objects displayed. The motivation to combine these references would have been to improve superimposition of the object onto the environment for ease of interaction an input by the user (see Kashitani at least at [0008]-[0011]). Therefore, a person of ordinary skill in the art would have been motivated to combine the prior art to achieve the claimed invention and there would have been a reasonable expectation of success.
Regarding claim 2, Han in view of Kashitani discloses the system according to claim 1 (see above), wherein the navigation content item, comprising the plurality of graphical elements (Kashitani at FIGS. 10A-10E illustrating multiple graphical objects provided on the displayed navigation projection at user’s feet [0122]-[0123] and [0131] and [0145]-[0149] describing multiple elements displayed such as E-mail, Navigation, Game etc.), comprises at least one ring displayed on the ground and centered around the user (Kashitani at FIGS. 10A-10E illustrating multiple graphical objects [0122]-[0123] and [0131] and [0145]-[0149] provided in an annular manner in relationship to the user; Han at FIG. 6, and 610 with user 611 in the center of the circular navigator 615 at [0091]-[0093] and FIGS. 9-13 at [0113]-[0118] and [0127]-[0133]), the plurality of graphical elements being distributed angularly on the at least one ring, one next to the other (Kashitani at FIGS. 10A-10E illustrating multiple graphical objects [0122]-[0123] and [0131] and [0145]-[0149] one distributed next to the other at an angle of the ring formed around the user; Han at FIG. 6, and 610 with user 611 the navigator 615 with objects next to one another distributed in an angular manner at [0091]-[0093] and FIGS. 9-13 at [0113]-[0118] and [0127]-[0133]).
Regarding claim 3, Han in view of Kashitani discloses the system according to claim 2 (see above), wherein, when the user (Kashitani FIGS. 10B-10D and [131] describing user) has come to rest on a graphical element corresponding to a deployment function (Kashitani at FIGS. 10A-10E illustrating multiple graphical objects [0122]-[0123] and [0131] when operator selects an option at FIG. 10C and [0145]-[0149]; Han, gaze determination of FIG. 11, operation 1111 at [0119]-[0126] and execution in FIGS. 12-13 at [0128]-[0133]) c), within a navigation content item, called a parent (Kashitani at FIG. 10B, boxes labeled EMAIL, NAVIGATION, GAME at [0122]-[0123] and [0131] and [0145]-[0149] which are navigation items/objects; Han at FIGS. 6, 9, 12-13 at [0113]-[0114] and [0128]-[0133] with objects such as 935, 1211, or object within line of sight 1311), the launcher displays another navigation content item, called a child dependent on the parent content item (Kashitani at FIGS. 10A-10E illustrating multiple graphical objects [0122]-[0123] and [0131] and [0145]-[0149] with 10C-D illustrating a concentric ring around the first ring dependent on the selection of the first object by OP of the user; Han at FIGS. 12-13 at [0113]-[0114] and [0128]-[0133] secondary objects such as thumbnails 1223 or additional information 1235,1335 related to the original object), and comprising a second ring, concentric to the at least one ring (Kashitani at FIGS. 10A-10E illustrating multiple graphical objects [0122]-[0123] and [0131] and [0145]-[0149] with 10C-D illustrating a second concentric ring around the first ring dependent on the selection of the first object), or a line, radial to the at least one ring (Han at FIGS. 12-13 at [0113]-[0114] and [0128]-[0133] secondary objects such as thumbnails 1223 or additional information 1235,1335 related to the original object and formed in relation to the object in a linear manner; or alternatively, Kashitani at FIGS. 10A-10E illustrating multiple graphical objects [0122]-[0123] and [0131] and [0145]-[0149] with 10C-D illustrating a second concentric ring around the first ring dependent on the selection of the first object), and comprising at least one other graphical element also corresponding to one of the functions a), b) or c), with a plurality of repetitions of parent-child navigation content items of successive generations (Kashitani execution/opening/deployment as shown in FIGS. 10A-10E and at [0122]-[1023] and [0145]-[0151]; Han at FIGS. 11-13 and [0119]-[0133] describing execution/opening/selection of objects such as 1233, 1235 and 1333, 1335 therein).
Regarding claim 5, Han in view of Kashitani discloses the system according to claim 1 (see above), wherein the interface proposes to the user (Kashitani FIGS. 10B-10D and [131] describing user), via the launcher, navigation content items that are contextual (Kashitani at FIGS. 10D-10E, [0123], [0149]-[0151] illustrating literal navigation content that is related to the location of the user and therefore contextual based on the location), taking into account the real environment of the user (Kashitani at FIGS. 1-9 and [0112]-[0114] describing real object determination/identification of objects in the environment; additionally FIGS. 10A-10E, determination of navigation in the environment [0123]-[0133]) and/or a context wherein the user is located (Kashitani FIGS. 10A-10E determination of navigation in the environment [0123]-[0133] and [0149]-[0151]), the interface determining the context by virtue of the at least one visual interaction device (DI) or any other device carried by the user and connected to the interface (Kashitani FIGS. 4-6, determination of positioning in a global coordinate system at [0087]-[0101] along with environment map building unit [0115]-[0120] to output a generated image [0122]-[0130] with superimposed images), providing it with information used by the interface (Kashitani FIGS. 1-6, determination of positioning in a global coordinate system at [0087]-[0101] along with environment map building unit [0115]-[0120] to output a generated image [0122]-[0130] the HMD with display D presents the user with the output image including superimposed images and FIGS. 10A-10E with clear navigation information provided to the user through the interface), among them at least:
receiving images captured by the at least one visual interaction device (DI), optionally with image recognition and/or detection of electronic codes (barcode, QR codes, etc.) (Kashitani FIGS. 1-6 and imaging unit 102 at [0071]-[0077]);
geolocation (Kashitani FIGS. 1-6, [0070]-[0080] with self-position detection unit; determination of positioning in a global coordinate system at [0087]-[0101] along with environment map building unit 150 [0115]-[0120] to output a generated image [0122]-[0130]);
detecting various physical values by various types of sensors (Kashitani [0132] and [0147] and [0151] imaging device using sensors capable of detecting gradient and tilt and rotation); and
receiving information provided by connected objects present in the real environment of the user and able to communicate with the at least one processor executing the interface (Kashitani FIG. 8, identification of the object and presentation of the object name at [0104]-[0110]).
Regarding claim 6, Han in view of Kashitani discloses a system according to claim 5 (see above), wherein the interface is configured to define at least one list of applications and/or content items or documents to be displayed, in particular, as a function of the real environment wherein the user is located (Kashitani at FIGS. 10A-10E navigation determination at [0122]-[0123] and [0149]-[0151] in view of FIGS. 4-6 determination of location and objects in environment at [0077]-[0100]), the list of applications and/or content items or documents potentially being updated on request from the user via the selection of a dedicated graphical element when they are not part of the selection displayed by the interface (Kashitani FIGS. 10A-10E, [0123], [0146], [0149] and FIG. 10C the selection of navigation, which opens the next row of objects including determination of directions to a convenient store, nearest station, dining, home as depicted), in order to obtain a wider set of functions, the at least one visual interaction device (DI) then loading in memory the data necessary for this update (Kashitani FIGS. 1-6, 9 and 10A-10E at [0073]-[0079], [0119]-[0120] and [0127] describing environmental map storage unit 152, new environmental mapping information built stored in a hard disk or semiconductor memory; such information would need to be updated based on the location of the user).
Regarding claim 7, Han in view of Kashitani discloses system according to claim 6 (see above), wherein the at least one visual interaction device (DI) is configured to update the interface (Kashitani at FIGS. 10A-10E illustrating multiple graphical objects [0122]-[0123] and [0131] and [0145]-[0149] with 10C-D illustrating object selection by OP of the user e.g., navigation button; Han at FIGS. 12-13 at [0113]-[0114] and [0128]-[0133]), with the user's selection of an element dedicated to this update triggering connection and synchronization of the at least one visual interaction device (DI) to the at least one processor containing a set of parameters, applications, content items or documents loadable into memory by the at least one visual interaction device (DI) (Kashitani FIGS. 1-6, 9 and 10A-10E at [0073]-[0079], [0119]-[0120] and [0127] describing environmental map storage unit 152, new environmental mapping information built stored in a hard disk or semiconductor memory; such information would need to be updated based on the location of the user, selection of the navigation elements in FIGS. 10A-10E as understood by one of ordinary skill would trigger the updated information; additionally Han at [0037] capability of information downloading remotely as known in the art) in order to render the at least one visual interaction device and the interface compatible with the immediate needs of the user (Kashitani FIGS. 10A-10E, [0123], [0146], [0149] and FIG. 10D the selection of navigation and data associated with selection of various destinations).
Regarding claim 9, Han in view of Kashitani discloses a method (Han at FIGS. 5, 8, 11, generally in view of FIGS. 6, 9, and 12-13 and [0119]-[0133]) comprising executing a human machine interface (FIGS. 5-13 generally describing navigator at [0051] and [0085]-[0093]) via the human machine interface system of claim 1 (see above).
Regarding claim 11, Han discloses a human-machine interface system (FIGS. 1-4, electronic device 100 at [0033]-[0040] and [0054]-[0055]) comprising:
at least one visual interaction device (FIGS. 1-4, head mounted display 200 with electronic device 100 at [0066]-[0071] and [0073]-[0076] which has a display 131 that has a touch input and user input unit 120 at [0040]-[0046]) comprising:
an image capture device (FIGS. 1-3 and camera module 170 at [0034], [0057], [0059], [0108] and [0122]);
a display (Abstract and FIGS. 1-4 display 131 at [0041]-[0046] and [0059] and [0064]); and
at least one processor (FIGS. 1-5 and controller 180 at [0034], [0041], [0047], [0050] and [0059]-[0065]) executing an interface (FIGS. 1-5 and visual output on display at [0041], GUI or user interface at [0032] and [0042]-[0043] and [0051] displayed by the display 131) for a user to interact (FIGS. 1-5 and [0040]-[0041] and [0046]-[0052] and user input 120 with interaction and inputs described therein) with the at least one visual interaction device (FIGS. 1-4, head mounted display 200 with electronic device 100 at [0066]-[0071] and [0073]-[0076]),
wherein the system (100) presents the user (FIG. 6 and user 611) with an at least partially virtual content item (FIGS. 6-8, [0103]-[0110] as presented in the display 131 to the user, content can be virtual or augmented reality at [0081], [0105] and [0121]), wherein the at least one visual interaction device (FIGS. 1-4, head mounted display 200 with electronic device 100) is configured to:
the interface configured to:
track a gaze of the user (FIGS. 11-13, describing movement of user’s focus/gaze in operations 1103-1109 to determine whether user’s focus is maintained [0119]-[0126]);
detect when the gaze of the user is oriented toward a ground area of the environment (FIG. 6 illustrating 615 with navigator at the feet around the user as disclosed at [0091]-[0093], generally FIG. 9 and [0112]-[0114]; and FIG. 8 at [0103]-[0104] operation 801 displaying the navigator according to the user’s gaze information); and
start a launcher (FIG. 8 at [0103]-[0104] operation 801 displaying the navigator according to the user’s gaze information) that is presented to the user via the at least one visual interaction device (FIG. 8, [0103]-[0111] describing presentation via the device 100 of navigator and content, FIG. 9, navigators 910, 920, 930, 940 at [0112]-[0114]), the launcher in the form of at least one navigation content item (FIG. 9, navigators 910, 920, 930, 940 at [0112]-[0114]) comprising a plurality of graphical elements (FIGS. 6, 9, 12, 13, e.g., object information 623 at [0093]-[0094], FIG. 9 and object information 935 and [0112]-[0114]) that the user is able to scroll through responsive to the gaze of the user (FIGS. 9-13, and [0103]-[0111] describing object selection at operation 807 therein; further describing updating the visible objects by determining the direction of the head and head-tracking at operation 803 and updating the visible objects on the display (e.g., scrolling through) accordingly) and that are representative of a function to which they correspond (FIGS. 11-13 and operations 1107-1111 and [0124]-[0129] and executing a function associated with the navigator and object at [0131]-[0133] with call, share, and view functions 1335);
detect movement of the gaze of the user across the plurality of graphical elements (FIGS. 11-13, describing movement of user’s focus/gaze in operations 1103-1109 to determine whether user’s focus is maintained [0119]-[0126]);
select a graphical element of the plurality of graphical elements (FIGS. 11-13, operation 1109-1111, determine whether user’s focus/gaze is maintained for a predetermined amount of time and executing the associated function with the object [0119]-[0126], further at least illustrating additional functions at FIG. 13 and additional information 1335 at [0131]-[0133]) responsive to the gaze of the user resting on the graphical element for a pre-determined duration of time (FIGS. 11-13, operation 1109, determine whether user’s focus/gaze is maintained for a predetermined amount of time [0119]-[0126]); and
trigger a function to which the selected graphical element corresponds (FIGS. 11-13, operation 1109-1111, determine whether user’s focus/gaze is maintained for a predetermined amount of time and executing the associated function with the object [0119]-[0126], further at least illustrating additional functions at FIG. 13 and additional information 1335 at [0131]-[0133]) the triggered function including one or more of a set of available functions comprising at least:
execute an application available via the interface and executable on the at least one visual interaction device (FIGS. 11-13; operation 1111 at [0126] for executing a function associated with the object information, such as playback of a moving image by the controller 180; additional information/functions 1335 that are executable at [0131]-[0133], the applications including call, share, view function, for example to the call function would be to call the person/contact, share would be send the location to the contact/counterpart);
open at least one content item or document stored in a non-transitory computer-readable storage device accessible via the interface (FIGS. 1-5 and 8-13 describing inputted/outputted data including files such as videos, images photos, audios, etc. at [0050] being stored in a memory 150, and retrieving the thumbnail images at least at [0110] and [0121]-[0126] to be displayed, operation 1111 of FIG. 11 displays/opens a list of executable functions associated with a contact at [0126]) for example gaze at 1331, causing 1333 to open at [0130]-[0133]); and
deploy other graphical elements in the launcher across which the user can move their gaze to trigger one or more of the set of available functions when it comes to rest thereupon for the pre-determined duration of time (FIGS. 8-13 and [0111]-[0114] describing gaze on 935 creating a new navigator with different objects in the content different than the original; further at [0121]-[0133] the operations such as 1111 performable on these new objects accordingly).
However, although Han discloses augmented reality ([0081], [0105] and [0121]), Han does not explicitly disclose capture an environment of the user to provide the environment to the interface, and generate a virtual mesh based, at least in part, on the environment.
In the same field of endeavor, Kashitani discloses a head mounted display ([0064]) with a display device D capable of imaging an image captured by a camera ([0064]) to capture an environment of the user to provide the environment to the interface (FIGS. 1-3, [0064]-[0068] describing capturing the environment 1 by the camera mounted on the head and outputs the image to the display device D of the head mounted display HMD), and generate a virtual mesh based, at least in part, on the environment (FIGS. 1-3, [0064]-[0068] the display, through an image processing device 100 display superimposition display data in a superimposed manner on actual space by optically compositing the superimposition display data within the visual field of the user).
Before the effective filing date, it would have been obvious to a person of ordinary skill in the art to modify the head mounted device for interaction by the user with displayed objects within the user’s visual field of Han to incorporate the real environment image capture and displaying as disclosed by Kashitani because the references are within the same field of endeavor, namely, virtual reality devices capable of user interaction with the objects displayed. The motivation to combine these references would have been to improve superimposition of the object onto the environment for ease of interaction an input by the user (see Kashitani at least at [0008]-[0011]). Therefore, a person of ordinary skill in the art would have been motivated to combine the prior art to achieve the claimed invention and there would have been a reasonable expectation of success.
Regarding claim 12, it is similar in scope to claim 2 above; therefore, claim 12 is similarly analyzed and rejected.
Regarding claim 13, it is similar in scope to claim 3 above; therefore, claim 13 is similarly analyzed and rejected.
Regarding claim 15, it is similar in scope to claim 5 above; therefore, claim 15 is similarly analyzed and rejected.
Regarding claim 16, it is similar in scope to claim 6 above; therefore, claim 16 is similarly analyzed and rejected.
Regarding claim 17, it is similar in scope to claim 7 above; therefore, claim 17 is similarly analyzed and rejected.
Regarding claim 19, Han discloses a method (Abstract and FIGS. 5, 8, and 11, generally and at [0080]-[0090]) comprising:
tracking, via a visual interaction device (FIGS. 1-4, head mounted display 200 with electronic device 100), a gaze of a user (FIGS. 5-9, and 11-13, and [0087] at operation 511 and [0097]-[0099], [0108] at operation 807, [0113]-[0114], and [0120]-[0122] operation 1103);
detecting when the gaze of the user is oriented past a pre-defined orientation boundary (FIG. 6 illustrating 615 with navigator at the feet around the user as disclosed at [0091]-[0093], generally FIG. 9 and [0112]-[0114]; and FIG. 8 at [0103]-[0104] operation 801 displaying the navigator according to the user’s gaze information; knowing when the gaze is at or within the bounds and focused on an object);
presenting a launcher (FIGS. 6, 9, 12-13 e.g., navigator 615 and [0092]-[0093] navigators 910, 920, 930, 940 at [0113]-[0114] and [0122]-[0132]) comprising a plurality of graphical elements (FIGS. 6, 9, 12, 13, e.g., object information 623 at [0093]-[0094], FIG. 9 and object information 935 and [0112]-[0114]) to the user via a display (Abstract and FIGS. 1-4 display 131 at [0041]-[0046] and [0059] and [0064]) of the visual interaction device (FIG. 8, [0103]-[0111] describing presentation via the device 100 of navigator and content, FIG. 9, navigators 910, 920, 930, 940 at [0112]-[0114] and FIGS. 1-5 with display 131 at [0041]-[0046] and [0078]);
detecting when a gaze of the user comes to rest on one of the plurality of graphical elements for a pre-determined period of time (FIGS. 11-13, operation 1109, determine whether user’s focus/gaze is maintained for a predetermined amount of time [0119]-[0126]); and
executing a pre-defined function responsive to the detection (FIGS. 11-13, operation 1109-1111, determine whether user’s focus/gaze is maintained for a predetermined amount of time and executing the associated function with the object [0119]-[0129], further at least illustrating additional functions at FIG. 13 and additional information 1335 at [0131]-[0133]).
However, although Han discloses augmented reality ([0081], [0105] and [0121]), Han does not explicitly disclose capturing an environment of the user via the visual interaction device; providing the environment to an interface of the visual interaction device; generating a virtual mesh of the environment.
In the same field of endeavor, Kashitani discloses a head mounted display ([0064]) with a display device D capable imaging an image captured by a camera ([0064]) by capturing an environment of the user via the visual interaction device (FIGS. 1-3, [0064]-[0068] describing capturing the environment 1 by the camera mounted on the head and outputs the image to the display device D of the head mounted display HMD); providing the environment to an interface of the visual interaction device (FIGS. 1-3, [0064]-[0068] describing capturing the environment 1 by the camera mounted on the head and outputs the image to the display device D of the head mounted display HMD); generating a virtual mesh of the environment (FIGS. 1-3, [0064]-[0068] the display, through an image processing device 100 display superimposition display data in a superimposed manner on actual space by optically compositing the superimposition display data within the visual field of the user).
Before the effective filing date, it would have been obvious to a person of ordinary skill in the art to modify the head mounted device for interaction by the user with displayed objects within the user’s visual field of Han to incorporate the real environment image capture and displaying as disclosed by Kashitani because the references are within the same field of endeavor, namely, virtual reality devices capable of user interaction with the objects displayed. The motivation to combine these references would have been to improve superimposition of the object onto the environment for ease of interaction an input by the user (see Kashitani at least at [0008]-[0011]). Therefore, a person of ordinary skill in the art would have been motivated to combine the prior art to achieve the claimed invention and there would have been a reasonable expectation of success.
Regarding claim 20, it is similar in scope to claim 5 above; therefore, claim 20 is similarly analyzed and rejected.
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Han in view of Kashitani as applied to claim 3 above, and further in view of Muta et al., US 2021/0357023 A1 (hereinafter “Muta”).
Regarding claim 4, Han in view of Kashitani discloses the system according to claim 3 (see above), wherein, when the user is scrolling through a child navigation content item without coming to rest on one of the graphical elements that it contains and moves their gaze to a parent navigation content item on which it depends potentially across a plurality of parent-child generations (Han at FIG. 11 at [0119]-[0126] and operations 1103-1107 describing opening the thumbnail when the user’s gaze/focus remains on an identified object, and at operation 1109 when it is determined that the user’s focus is or is not maintained repeating the steps of 1103-1107 or executing the function at 1111),
However, Han in view of Kashitani does not explicitly disclose the launcher, after a determined time period, withdraws the child navigation content item(s) by canceling their display(s) on the at least one visual interaction device (DI) and displays only the parent navigation content item on which the user is fixing their gaze.
In the same field of endeavor, Muta discloses the launcher, after a determined time period, withdraws the child navigation content item(s) by canceling their display(s) on the at least one visual interaction device (DI) and displays only the parent navigation content item on which the user is fixing their gaze (FIGS. 3-9, and [0083]-[0087] describing relationship of first object 300b and second object 500 therein, [0090]-[0094] removal of second object 500 after line of sight is not detected for a second length of time; FIGS. 10-12 at [0100]-[0117] steps S103-S108, S202-S207 describing method of determining line of sight on first object 300b and generating second object 500, and [0118]-[0127] steps S302-2309 describing removal of second object when line of site off of the object exceeds a threshold time).
Before the effective filing date, it would have been obvious to a person of ordinary skill in the art to modify the user interface device with gaze detection and input of Han in view of Kashitani to incorporate the removal of a secondary object upon removal of the gaze for a predetermined amount of time as disclosed by Muta because the references are within the same field of endeavor, namely, gaze input determination user interface display devices. The motivation to combine these references would have been to improve determination of intentionality of inputs and focus on an object by the user (see Muta at least at [0005]-[0006]). Therefore, a person of ordinary skill in the art would have been motivated to combine the prior art to achieve the claimed invention and there would have been a reasonable expectation of success.
Regarding claim 14, it is similar in scope to claim 2 above; therefore, claim 12 is similarly analyzed and rejected.
Claims 8, 10, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Han in view of Kashitani as applied to claims 7 and 17 above, and further in view of Clausen et al., US 2019/0279424 A1 (hereinafter “Clausen”).
Regarding claim 8, Han in view of Kashitani discloses a system according to claim 7 (see above).
However, Han in view of Kashitani does not explicitly disclose wherein the system comprises a plurality of interaction devices (DI) connected to each other, preferably securely to form a shared device cluster, at least one of the plurality of interaction devices (DI) being configured to share by transmission instructions relating to content items to be displayed on the interface of the any other device depending on actions carried out via the interface, so that all of the plurality of interaction devices (DI) display the same elements as a result of actions or selections performed via the interface of the at least one of the plurality of interaction devices (DI).
In the same field of endeavor, Clausen discloses collaborative augmented reality system (Abstract) wherein the system comprises a plurality of interaction devices (DI) connected to each other (FIG. 13, at [0219] multiple head-mounted wearable devices 1000), preferably securely to form a shared device cluster ([0027] session may be connected into one or more shared sessions with non-collocated users and uses secure web applications at [0181]-[0194]), at least one of the plurality of interaction devices (DI) being configured to share by transmission instructions relating to content items to be displayed on the interface of the any other device depending on actions carried out via the interface ([0065]-[0067] gestures indicating transmission of specific instructions or identification of content being displayed), so that all of the plurality of interaction devices (DI) display the same elements as a result of actions or selections performed via the interface of the at least one of the plurality of interaction devices (DI) ([0065]-[0067] gestures indicating transmission of specific instructions or identification of content being displayed visible to other users/participants via highlighting or marking)).
Before the effective filing date, it would have been obvious to a person of ordinary skill in the art to modify the head mounted display device capable of user input of Han in view of Kashitani to incorporate the collaborative element for head mounted displays as disclosed by Clausen because the references are within the same field of endeavor, namely, head mounted display with virtual interfaces capable of input. The motivation to combine these references would have been to improve immersive techniques for collaborative models at true scale enabling accurate visualization for the users (see Clausen at least at [0003]-[0004]). Therefore, a person of ordinary skill in the art would have been motivated to combine the prior art to achieve the claimed invention and there would have been a reasonable expectation of success.
Regarding claim 10, Han in view of Kashitani discloses the human-machine interface system according to claim 8 (see above), wherein the plurality of visual interaction devices (DI) each comprises an image capture device (Han, FIGS. 1-3 and camera module 170 at [0034], [0057], [0059], [0108] and [0122]), a display (Han, Abstract and FIGS. 1-4 display 131 at [0041]-[0046] and [0059] and [0064]), and at least one processor (Han, FIGS. 1-5 and controller 180 at [0034], [0041], [0047], [0050] and [0059]-[0065]) running an interface (Han, FIGS. 1-5 and visual output on display at [0041], GUI or user interface at [0032] and [0042]-[0043] and [0051] displayed by the display 131) for user interaction (Han, FIGS. 1-5 and [0040]-[0041] and [0046]-[0052] and user input 120 with interaction and inputs described therein) with the at least one visual interaction device (DI) (Han, FIGS. 1-4, head mounted display 200 with electronic device 100 at [0066]-[0071] and [0073]-[0076]),
the system presenting the user (Han, FIG. 6 and user 611) with an at least partially virtual content item (Han, FIGS. 6-8, [0103]-[0110] as presented in the display 131 to the user, content can be virtual or augmented reality at [0081], [0105] and [0121]), in particular, for augmented reality (Han, FIGS. 5-6 and electronic device producing virtual reality or augmented reality at [0081], [0105] and [0121]),
the plurality of visual interaction devices (DI) (Han, FIGS. 1-4, head mounted display 200 with electronic device 100) capturing the real environment of the user in order to provide it to the interface (Kashitani FIGS. 1-3, [0064]-[0068] describing capturing the environment 1 by the camera mounted on the head and outputs the image to the display device D of the head mounted display HMD), which generates a mesh of the real environment (Kashitani FIGS. 1-3, [0064]-[0068] the display, through an image processing device 100 display superimposition display data in a superimposed manner on actual space by optically compositing the superimposition display data within the visual field of the user) and tracks the user's gaze (Han, FIGS. 5-9, and 11-13, and [0087] at operation 511 and [0097]-[0099], [0108] at operation 807, [0113]-[0114], and [0120]-[0122] operation 1103), the interface being connected to each of the users' visual interaction devices (DI) to enable them to share data with each other via the at least one processor and display content to them (Clausen, [0065]-[0069] gestures indicating transmission of specific instructions or user ability for identification of content being displayed visible to other users)), which is synchronized on various securely communicating visual interaction devices (DI) (Clausen, [0065]-[0069] user are notified of parts that are being highlighted synchronously), the system comprising a non-transitory computer-readable storage device (Clausen, [0194] storage of models [0210]-[0213] [0216] describing storage devices 1120 of FIGS. 11-12;) containing objects, documents and 3D maps (Clausen, [0194] storage of models [0210]-[0213] [0216] describing storage devices 1120 of FIGS. 11-12;) (maps of virtual environments established from real environments and representative of maps of real sites or places previously recorded with various levels of detail as to their topography and/or their geolocation and/or various technical information relating to physical elements present in these sites or places) (Kashitani at FIGS. 4-7, and 8-12; [0011] and [0026] describing environment map further at [0069]-[0073] and [0103]-[0112] and [0127] and [0007] and [0075]-[0078] and [0082][0085], environment map storage unit 152 and [0119] and [0161]), the interface being configured to track actions of each of the users in the virtual environment (Clausen at [0041]-[0044] and [0071]-[0089] describing collaborative tools; sharing of designs at [0138] and creating collaborative space [0145]) thus presented to the users of the communicating visual interaction devices to which the interface proposes a set of content items, objects, or documents sharable with the users, so that the tracking of their respective actions enables them to place these various content items, objects or documents within the virtual environment (Clausen at [0041]-[0044] and [0071]-[0089] describing collaborative tools and CAD drawing therein; sharing of designs at [0138] and creating collaborative space [0145], in view of FIG. 13 and [0219]), the system also being connected to at least one sensor or communicating visual interaction device of a user present within the real environment thus simulated in order to provide the interface (Kashitani at [0141]-[0151] and recognize a hand imaged from the user and FIGS. 10A-10E with user foot as OP as disclosed at [0131] and [0140]-[0148] and [0152]-[0153]; in a collaborative environment of shared objects captured by sensors and cameras of Clausen and [0065]-[0068] and [0127]-[0130] and [0202] [0206] and [0219] and FIG. 13, the combination of which would be obvious to one of ordinary skill to properly communicate interaction in the collaborative environment), in real time or predicted according to a schedule (Kashitani at [0141]-[0151] and recognize a hand imaged from the user and FIGS. 10A-10E with user foot as OP as disclosed at [0131] and [0140]-[0148] and [0152]-[0153]), with information relating to physical parameters measured in real time in the real environment or predicted for a later time or date (Kashitani at [0140]-[0151] and recognize and displayed a hand or foot imaged from the user and providing mapping and updating in real time and object prediction of objects and predictive models in the real environment at FIGS. 5-8 and [0079]-[0093] and FIGS. 10A-10E with user foot as OP as disclosed at [0131] and [0140]-[0148] and [0152]-[0153]), in order to provide the users of the communicating devices with the actual physical information present or predictable within the real environment (Kashitani at [0140]-[0151] and recognize and providing mapping and updating in real time predictive models in the real environment at FIGS. 5-8 and [0079]-[0093] and FIGS. 10A-10E with user foot as OP as disclosed at [0131] and [0140]-[0148] and [0152]-[0153]).
Regarding claim 18, it is similar in scope to claim 8 above; therefore, claim 18 is similarly analyzed and rejected.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Richman et al., US 2022/0107685 A1;
Lee et al., US 2015/0355709 A1;
Chachek et al., US 2020/0302510 A1;
Rainisto, US 2016/0274765 A1;
Ross, US 9,392,212 A1;
Any inquiry concerning this communication or earlier communications from the examiner should be directed to