DETAILED ACTION
This action is in response to communication filed on 25 September 2024. Claims 1-3 are pending in the application and have been considered below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim 2 is rejected under 35 U.S.C. 102(a)(2) as being anticipated by KOCIENDA et al. (US20230280822A1).
As to claim 2, KOCIENDA teaches a method comprising:
projecting an ephemeral user interface onto a hand palm, the ephemeral user interface including a first set of affordances (see figs. 10A-16, par. 0124, wherein the wearable multimedia device 101 can be configured to detect objects that are positioned in the projection area. For example, the wearable multimedia device 101 can obtain sensor data from one or more sensors (e.g., the camera/video subsystems 820, environment sensor(s) 817, depth sensor(s) 814, etc.), and based on the sensor data, detect whether the user has positioned a hand in the projection area. Further, based on the sensor data, the wearable multimedia device 101 can distinguish between the user positioning her right hand in the projection area and the user positioning her left hand in the projection area; see also par. 0125, wherein upon detecting that the user's left hand is positioned in the projection area of the projector subsystem, the wearable multimedia device can perform one or more first operations. As an example, the wearable multimedia device can execute one or more first applications, present one or more first VIs on the surface of the user's left hand, and/or perform one or more first functions or tasks. Further, upon detecting that the user's right hand is positioned in the projection area of the projector subsystem, the wearable multimedia device can perform one or more second operations. As an example, the wearable multimedia device can execute one or more second applications, present one or more second VIs on the surface of the user's right hand, and/or perform one or more second functions or tasks; as taught by KOCIENDA);
detecting a change in depth of the hand palm relative to a depth sensor; and responsive to the change in depth, projecting a second set of affordances in the ephemeral user interface (see par. 0126, wherein referring to FIG. 10A, a projector subsystem 832 can be configured to project light into a projection area 1002 (represented as a dashed outline). Although the projection area 1002 is depicted in FIG. 10A as a rectangular region of space, in practice, the projection area 1002 can be any shape (e.g., polygonal, curved, and/or a combination thereof). In this example, the user has positioned her left hand 1004 within the projection area 1002. The wearable multimedia 101 detects the presence of the user's left hand 1004 in the projection area 1002 (e.g., using the camera/video subsystems 820, environment sensor(s) 817, depth sensor(s) 814, etc.), and in response performs one or more first operations (“Operation(s) A”); see also par. 0127, wherein In the example shown in FIG. 10B, the user has positioned her right hand 1006 within the projection area 1002. The wearable multimedia 101 detects the presence of the user's right hand 10006 in the projection area 1002 (e.g., using the camera/video subsystems 820, environment sensor(s) 817, depth sensor(s) 814, etc.), and in response performs one or more second operations (“Operation(s) B”); as taught by KOCIENDA).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 3 are rejected under 35 U.S.C. 103 as being unpatentable over LU (WO2020124995A1) in view of KOCIENDA et al. (US20230280822A1).
As to claim 1, LU teaches a method comprising:
detecting, using a first machine learning model, a three-dimensional (3D) image of a hand based on depth images of the hand (see page 5, ll. 25-31, wherein in one implementation of the embodiments of the present disclosure, the acquiring a palm image of the user includes: acquiring a user image, the user image including the palm of the user and a background outside the palm; and inputting the user image into In the first deep learning model, the position data of the palm detection frame is obtained; the palm image of the user is intercepted from the user image based on the position data of the palm detection frame; as taught by LU);
predicting, using a second machine learning model, keypoints on the hand (see page 14, ll. 12-22, wherein in one embodiment, the palm correction image determination module 420 is configured to input the palm image into the second deep learning model to obtain the position coordinates of multiple palm joint points and the identification of the palm joint point corresponding to each position coordinate Based on the identification of the palm joint point, select the position coordinates of the first target joint point and the position coordinates of the second target joint point from the position coordinates of the palm joint points; the position of the first target joint point The coordinates are transformed to the position coordinates of the first preset joint point, and the position coordinates of the second target joint point are transformed to the position coordinates of the second preset joint point to obtain a correction matrix; based on the correction matrix, the palm The image undergoes coordinate transformation to obtain a palm correction image; as taught by LU);
estimating, based on the keypoints and depth images, a normal to a palm of the hand;
computing a difference between the estimated normal and a resting normal (see page 7, ll. 1-15, wherein in one implementation of the embodiments of the present disclosure, the determining the normal vector of the corrected palm image includes: inputting the palm correction image into a third deep learning model to obtain the palm of the palm correction image Normal vector. In an embodiment, the third deep learning model may be a trained neural network model, or may also be another trained model. The training process of the third deep learning model can be: select the palm image training sample, use the palm image training sample as the input of the third deep learning model, and use the normal vector of the palm in the palm image as the output of the third deep learning model. Three neural network models are trained to obtain the trained third deep learning model; In an embodiment, when the palm correction image is input to the third deep learning model, the third deep learning model may output the normal vector of the palm in the palm correction image, and use the output normal vector as the target normal vector. In an embodiment, the palm normal vector may be a vector perpendicular to the plane of the palm. In an embodiment, the palm normal vector may be a unit vector perpendicular to the plane of the palm; as taught by LU);
LU does not expressly teach projecting the difference into a two-dimensional (2D) plane of an ephemeral user interface projected on the user’s hand palm; focusing an underlying cursor on an affordance in the ephemeral user interface based on a location of the projected difference in the 2D plane; detecting a gesture made by the user’s hand; and responsive to detecting the gesture, selecting the affordance.
In similar field of endeavor, KOCIENDA teaches:
projecting the difference into a two-dimensional (2D) plane of an ephemeral user interface projected on the user’s hand palm (see figs. 10A-16, par. 0126, wherein referring to FIG. 10A, a projector subsystem 832 can be configured to project light into a projection area 1002 (represented as a dashed outline). Although the projection area 1002 is depicted in FIG. 10A as a rectangular region of space, in practice, the projection area 1002 can be any shape (e.g., polygonal, curved, and/or a combination thereof). In this example, the user has positioned her left hand 1004 within the projection area 1002. The wearable multimedia 101 detects the presence of the user's left hand 1004 in the projection area 1002 (e.g., using the camera/video subsystems 820, environment sensor(s) 817, depth sensor(s) 814, etc.), and in response performs one or more first operations (“Operation(s) A”); as taught by KOCIENDA);
focusing an underlying cursor on an affordance in the ephemeral user interface based on a location of the projected difference in the 2D plane; detecting a gesture made by the user’s hand; and responsive to detecting the gesture, selecting the affordance (see figs. 10A-16, pars. 0136-0152, e.g. par. 0149, wherein upon detecting that the user's left hand 1004 is positioned in the projection area 1002 of the projector subsystem 832, the wearable multimedia device 101 can automatically project an application selection VI 1404 on the user's left hand 1006. The application selection VI 1404 can display one or more applications that are available for execution by the wearable multimedia device 101 (e.g., using one or more icons, text labels, lists, menus, user interface elements, etc.). The user can browse the application selection VI 1404 to review the available applications, and select one or more of the applications for execution by the wearable multimedia device 101; see also fig. 0150, wherein the user can make a selection using her “free” hand (e.g., the hand opposite to the hand on which the application selection VI 1404 is projected). For example, referring to FIG. 14A, the user can tap or press a finger of her right hand 1006 onto the surface of her left hand 1004, such that the tip of the user's finger coincides with a particular application presented in the application selection VI 1404. In response to detecting this gesture (e.g., using the camera/video subsystems 820, environment sensor(s) 817, depth sensor(s) 814, etc.), the wearable multimedia device 101 can execute the selected application; see also par. 0151], wherein the user can also make a selection using the hand on which the application selection VI 1404 is projected. For example, referring to FIG. 14A, the user can make a selection by hovering a finger of her right hand 1006 over one application presented in the application selection VI 1404. Further, the user can confirm her selection by performing a gesture using her left hand 1004. As an example, while the user is hovering her finger of her right hand 1006 over the application, using her left hand 1004, the user can touch the tip of her thumb with the tip of another finger, such as her index finger. In response to detecting this gesture (e.g., using the camera/video subsystems 820, environment sensor(s) 817, depth sensor(s) 814, etc.), the wearable multimedia device 101 can execute the selected application; see also pars. 0104-0106; as taught by KOCIENDA).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the LU apparatus to include the teachings of KOCIENDA for projecting the difference into a two-dimensional (2D) plane of an ephemeral user interface projected on the user’s hand palm; focusing an underlying cursor on an affordance in the ephemeral user interface based on a location of the projected difference in the 2D plane; detecting a gesture made by the user’s hand; and responsive to detecting the gesture, selecting the affordance. Such a person would have been motivated to make this combination as this implementations can provide various technical benefits. For instance, these techniques allow the user to associate certain functions of the wearable multimedia device with each hand, such that she can interact with the wearable multimedia device in a more organized, consistent, predictable, and/or intuitive manner. Accordingly, the user can interact with the wearable multimedia device more quickly and efficiently, and is less likely to provide erroneous and/or unintended inputs to the wearable multimedia device. (see KOCIENDA, par. 0007).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over KOCIENDA et al. (US20230280822A1) in view of TYLER et al. (US20220365645A1).
As to claim 3, KOCIENDA teaches a method comprising:
projecting on a hand palm (see figs. 10A-16, par. 0124, wherein the wearable multimedia device 101 can be configured to detect objects that are positioned in the projection area. For example, the wearable multimedia device 101 can obtain sensor data from one or more sensors (e.g., the camera/video subsystems 820, environment sensor(s) 817, depth sensor(s) 814, etc.), and based on the sensor data, detect whether the user has positioned a hand in the projection area. Further, based on the sensor data, the wearable multimedia device 101 can distinguish between the user positioning her right hand in the projection area and the user positioning her left hand in the projection area; see also par. 0125, wherein upon detecting that the user's left hand is positioned in the projection area of the projector subsystem, the wearable multimedia device can perform one or more first operations. As an example, the wearable multimedia device can execute one or more first applications, present one or more first VIs on the surface of the user's left hand, and/or perform one or more first functions or tasks. Further, upon detecting that the user's right hand is positioned in the projection area of the projector subsystem, the wearable multimedia device can perform one or more second operations. As an example, the wearable multimedia device can execute one or more second applications, present one or more second VIs on the surface of the user's right hand, and/or perform one or more second functions or tasks; as taught by KOCIENDA);
detecting a change in depth of the hand palm relative to a depth sensor;
and responsive to the change in depth (see par. 0126, wherein referring to FIG. 10A, a projector subsystem 832 can be configured to project light into a projection area 1002 (represented as a dashed outline). Although the projection area 1002 is depicted in FIG. 10A as a rectangular region of space, in practice, the projection area 1002 can be any shape (e.g., polygonal, curved, and/or a combination thereof). In this example, the user has positioned her left hand 1004 within the projection area 1002. The wearable multimedia 101 detects the presence of the user's left hand 1004 in the projection area 1002 (e.g., using the camera/video subsystems 820, environment sensor(s) 817, depth sensor(s) 814, etc.), and in response performs one or more first operations (“Operation(s) A”); see also par. 0127, wherein In the example shown in FIG. 10B, the user has positioned her right hand 1006 within the projection area 1002. The wearable multimedia 101 detects the presence of the user's right hand 10006 in the projection area 1002 (e.g., using the camera/video subsystems 820, environment sensor(s) 817, depth sensor(s) 814, etc.), and in response performs one or more second operations (“Operation(s) B”); as taught by KOCIENDA).
KOCIENDA does not expressly teach a virtual stack of elements with a first element in the stack displaying a first value; replacing the first element with a second element of the stack displaying a second value.
In similar field of endeavor, TYLER teaches:
a virtual stack of elements with a first element in the stack displaying a first value (see figs. 5E1-5E10, par. 0347, wherein FIG. 5E1 shows a respective user-arranged page 5202 of a multipage home screen user interface (also referred to as “user-arranged home screen 5202”), including a plurality of application icons (e.g., application icons 5008 a-5008 k), and one or more widgets (e.g., widget 5022 j) and/or widget stacks (e.g., widget stack 5024 b) arranged in a preset layout (e.g., on a 6×4 grid for application icons, with placement locations for widgets and widget stacks specified in terms of full rows and half rows of the 6×4 grid). A widget stack includes multiple widgets corresponding to different applications and only a subset of the multiple widgets (e.g., one widget, two widgets, etc.) displayed at the placement location of the widget stack at a time. In the widget stack 5024 b, the currently displayed widget is widget 5022 j corresponding to an application “App 3”; as taught by TYLER);
replacing the first element with a second element of the stack displaying a second value (see par. 0349, wherein FIGS. 5E2-5E3 illustrate that, in response to detecting the swipe input by the contact 6100 at the location of the widget stack 5024 b (e.g., as shown in FIG. 5E2) and in accordance with a direction of the swipe input by the contact 6100, the device 100 replaces the currently displayed widget (e.g., widget 5022 g) with another widget in the stack (e.g., a widget that is next to the currently displayed widget in the stack (e.g., widget 5022 k), as shown in FIG. 5E3). In some embodiments, widget indicator icons 5330 are displayed adjacent to or at the location of the widget stack 5024 b when the currently displayed widget of the widget stack 5024 b is switched (e.g., due to manual switching in response a user input (e.g., the upward swipe input by the contact 6100), and/or due to automatic switching according to a schedule or a changed context, etc.). The widget indicator icons 5330 indicates the number of widgets included in the widget stack and a position of the currently displayed widget relative to other widgets in the widget stack. For example, in FIG. 5E2, widget indicator icon 5330 e is highlighted, indicating that widget 5022 g is the second widget in widget stack 5024 b; and in FIG. 5E3, after widget 5022 g is replaced with widget 5022 k in response to the swipe input by the contact 6100, widget indicator icon 5330 f is highlighted, indicating that widget 5022 k is the third widget and last widget in the widget stack 5024 b; as taught by TYLER).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the KOCIENDA apparatus to include the teachings of TYLER with a virtual stack of elements with a first element in the stack displaying a first value; replacing the first element with a second element of the stack displaying a second value. Such a person would have been motivated to make this combination as this implementation can provide various technical benefits, especially where there is a small display or projection area and a large number of items to be displayed. In this manner, the items are overlaid and will appear one by one when the user desires (see TYLER, pars. 0003-0005).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Publication Number
Filing Date
Title
US20230280867A1
2022-03-04
Laser projection on fingernail
US20220400235A1
2022-06-10
Dynamic Optical Projection With Wearable Multimedia Devices
US20180332211A1
2018-05-10
Wearable Multimedia Device and Cloud Computing Platform with Application Ecosystem
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KOOROSH NEHCHIRI whose telephone number is (408)918-7643. The examiner can normally be reached M-F, 9-5 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William L. Bashore can be reached on 571-272-4088. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KOOROSH NEHCHIRI/Examiner, Art Unit 2174
/WILLIAM L BASHORE/ Supervisory Patent Examiner, Art Unit 2174
Prosecution Insights