DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (2021/0166486) in view of Ibtehaz (2021/0279554).
Regarding claim 1, Kim teaches an electronic device comprising: memory (150; Fig 3) storing instructions (para [0070] Also, the memory 150 may store a program, data, and the like for constituting various types of screens that will be displayed in the display area of the display 120.); at least one sensor module (110; Fig 2, Fig 3) configured to detect a hand of a user (para [0077] If the user body detected from an image captured by the camera 110 is a hand,); and at least one processor (130; Fig 3) comprising processing circuitry; wherein the instructions, when executed by the at least one processor individually or collectively (para [0057]), cause the electronic device to: execute a first application configured to provide an augmented reality (AR) image to the user (para [0017] the electronic apparatus in which a display displays an image, rendering a virtual object; para [0047] The electronic apparatus 100 may provide the AR image 11 to the user using a display, ); select a first artificial intelligence (AI) model, based on execution of the first application (para [0059] When the user body is detected from the captured image, the processor 130 may estimate a plurality of joint coordinates for the detected user body using a pre-trained learning model (S230)); obtain first data related to the hand of the user from the at least one sensor module (para [0050] The electronic apparatus 100 may estimate 3D coordinates of the user body by using a pre-trained learning model even when only a portion of the user body (e.g., hand) is captured.); identify a position of the hand of the user (para [0050] The electronic apparatus 100 may estimate 3D coordinates of the user body by using a pre-trained learning model even when only a portion of the user body (e.g., hand) is captured. In addition, the electronic apparatus 100 may estimate an exact location) and a shape of joints in the hand by using the first AI model to calculate the first data (para [0059] the processor 130 may estimate motions, shapes, and predicted coordinates of the user body by inputting the extracted RGB data); based on the position of the hand of the user and the joints in the hand, identified using the first AI model, determine whether there is an interaction between the hand of the user and at least one object included in the AR image (para [0062] The processor 130 may control the display 120 to display the generated AR image. When the AR image is displayed on the display 120, the user may interact with the virtual object through the AR image. Specifically, the processor 130 may check whether the user body touches the virtual object based on the estimated joint coordinates, and perform an event corresponding to the object touch when the object touch is detected.); obtain second data related to the hand of the user from the at least one sensor module; and identify a position of the hand of the user and a shape of the joints in the hand to calculate the second data (para [0062] specifically, the processor 130 may check whether the user body touches the virtual object based on the estimated joint coordinates, and perform an event corresponding to the object touch when the object touch is detected.).
Kim fails to teach, select a second AI model, based on determination that there is an interaction between the hand of the user and at least one object included in the AR image; by using the second AI model to calculate the second data, and wherein the first AI model is configured to process a first calculation amount for a first time period, and the second AI model is configured to process a second calculation amount greater than the first calculation amount for the first time period; as claimed.
Ibtehaz teaches an apparatus comprising: select a first artificial intelligence (AI) model, based on execution of a first application (para [0008] The at least one processor may be further configured to execute the one or more instructions to verify the identified physical activity; para [0009] The at least one artificial intelligence learning model may include a first artificial intelligence learning model and a second artificial intelligence learning model, and the at least one processor may be further configured to execute the one or more instructions to: identify a first physical activity of the user by inputting the first type sensor data into the first artificial intelligence learning model;); select a second artificial intelligence (AI) model, based on second type of sensor data (para [0057] when the determined physical activity is included in the pre-defined physical activities, may input the second type sensor data into the second AI learning model to verify the determined physical activity in operation S320.); and wherein the first AI model is configured to process a first calculation amount for a first time period (para [0051] The first AI learning model may be trained by using, as the training data, the first type sensor data, which is collected from the first type wearable sensor when a person wearing the first type wearable sensor performs various physical activities), and the second AI model is configured to process a second calculation amount (para [0051] and the second AI learning model may be trained by using, as the training data, the second type sensor data, which is collected from the second type wearable sensor when a person wearing the second type wearable sensor performs various physical activities.).
It would have been obvious to one of ordinary skill in the art before the filing date of present application to have modified the device of Kim with the teachings of Ibtehaz, because it is well known to use different types of AI models for processing different types of data, in order to yield predictable results.
Kim and Ibtehaz fails to teach, the second calculation amount greater than the first calculation amount for the first time period; as claimed.
However, given that prior art Kim teaches first operation of identify a position of the hand of the user and a shape of joints in the hand to calculate first data; and further teaches second operation of determination that there is an interaction between the hand of the user and at least one object included in the AR image; it would be obvious that the second calculation amount with respect to the second operation will be greater than the first calculation amount with respect to the first operation, because the first operation corresponds to first data regarding position of hand and shape of the joints in the hand; while second operation corresponds to second data regarding interaction between the hand and virtual object. Thus, the added computation of virtual object would account for greater calculation amount.
Regarding claim 12, Kim teaches a method of an electronic device, the method comprising: executing a first application configured to provide an augmented reality (AR) image to a user (para [0017] the electronic apparatus in which a display displays an image, rendering a virtual object; para [0047] The electronic apparatus 100 may provide the AR image 11 to the user using a display, ); selecting a first artificial intelligence (AI) model, based on execution of the first application (para [0059] When the user body is detected from the captured image, the processor 130 may estimate a plurality of joint coordinates for the detected user body using a pre-trained learning model (S230)); obtaining first data related to the hand of the user from at least one sensor module (110; Fig 2, Fig 3; para [0050] The electronic apparatus 100 may estimate 3D coordinates of the user body by using a pre-trained learning model even when only a portion of the user body (e.g., hand) is captured.); identifying a position of the hand of the user (para [0050] The electronic apparatus 100 may estimate 3D coordinates of the user body by using a pre-trained learning model even when only a portion of the user body (e.g., hand) is captured. In addition, the electronic apparatus 100 may estimate an exact location) and a shape of joints in the hand by using the first AI model to calculate the first data (para [0059] the processor 130 may estimate motions, shapes, and predicted coordinates of the user body by inputting the extracted RGB data); based on the position of the hand of the user and the joints in the hand identified using the first AI model, determining whether there is an interaction between the hand of the user and at least one object included in the AR image (para [0062] The processor 130 may control the display 120 to display the generated AR image. When the AR image is displayed on the display 120, the user may interact with the virtual object through the AR image. Specifically, the processor 130 may check whether the user body touches the virtual object based on the estimated joint coordinates, and perform an event corresponding to the object touch when the object touch is detected.); obtaining second data related to the hand of the user from the at least one sensor module; and identifying a position of the hand of the user and a shape of the joints in the hand to calculate the second data (para [0062] specifically, the processor 130 may check whether the user body touches the virtual object based on the estimated joint coordinates, and perform an event corresponding to the object touch when the object touch is detected.).
Kim fails to teach, selecting a second AI model, based on determination that there is an interaction between the hand of the user and at least one object included in the AR image; by using the second AI model to calculate the second data, and wherein the first AI model is configured to process a first calculation amount for a first time period, and the second AI model is configured to process a second calculation amount greater than the first calculation amount for the first time period; as claimed.
Ibtehaz teaches a method comprising: selecting a first artificial intelligence (AI) model, based on execution of a first application (para [0008] The at least one processor may be further configured to execute the one or more instructions to verify the identified physical activity; para [0009] The at least one artificial intelligence learning model may include a first artificial intelligence learning model and a second artificial intelligence learning model, and the at least one processor may be further configured to execute the one or more instructions to: identify a first physical activity of the user by inputting the first type sensor data into the first artificial intelligence learning model;); selecting a second artificial intelligence (AI) model, based on second type of sensor data (para [0057] when the determined physical activity is included in the pre-defined physical activities, may input the second type sensor data into the second AI learning model to verify the determined physical activity in operation S320.); and wherein the first AI model is configured to process a first calculation amount for a first time period (para [0051] The first AI learning model may be trained by using, as the training data, the first type sensor data, which is collected from the first type wearable sensor when a person wearing the first type wearable sensor performs various physical activities), and the second AI model is configured to process a second calculation amount (para [0051] and the second AI learning model may be trained by using, as the training data, the second type sensor data, which is collected from the second type wearable sensor when a person wearing the second type wearable sensor performs various physical activities.).
It would have been obvious to one of ordinary skill in the art before the filing date of present application to have modified the method of Kim with the teachings of Ibtehaz, because it is well known to use different types of AI models for processing different types of data, in order to yield predictable results.
Kim and Ibtehaz fails to teach, the second calculation amount greater than the first calculation amount for the first time period; as claimed.
However, given that prior art Kim teaches first operation of identify a position of the hand of the user and a shape of joints in the hand to calculate first data; and further teaches second operation of determination that there is an interaction between the hand of the user and at least one object included in the AR image; it would be obvious that the second calculation amount with respect to the second operation will be greater than the first calculation amount with respect to the first operation, because the first operation corresponds to first data regarding position of hand and shape of the joints in the hand; while second operation corresponds to second data regarding interaction between the hand and virtual object. Thus, the added computation of virtual object would account for greater calculation amount.
Claim(s) 3-5, 14 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (2021/0166486) in view of Ibtehaz (2021/0279554) as applied to claim 1 and 12 above, and further in view of Ravasz et al. (2020/0388247).
Regarding claim 3, Kim and Ibtehaz teaches the electronic device as explained for claim 1 above.
Kim and Ibtehaz fails to teach, based on the position of the hand of the user and the joints in the hand, identified using the first AI model, determine a gesture of the hand of the user; identify whether the determined gesture is a first gesture; and based on the determined gesture being the first gesture, determine that the interaction is present; as claimed.
Ravasz teaches an artificial reality system, comprising: based on the position of the hand of the user and the joints in the hand, identified using the first AI model (para [0037] o detect the gesture(s), the artificial reality application may compare the motions, configurations, positions and/or orientations of hand 132 and/or portions of arm 134 to gesture definitions stored in a gesture library of artificial reality system 10, where each gesture in the gesture library may be each mapped to one or more actions.), determine a gesture of the hand of the user (para [0108] Similar to the examples described with respect to FIG. 3, based on the sensed data, gesture detector 424 analyzes the tracked motions, configurations, positions, and/or orientations of objects (e.g., hands, arms, wrists, fingers, palms, thumbs) of the user to identify one or more gestures performed by user 110. ); identify whether the determined gesture is a first gesture (para [0108] Gesture detector 424 may track movement, including changes to position and orientation, of the hand, digits, and/or arm based on the captured image data, and compare motion vectors of the objects to one or more entries in gesture library 430 to detect a gesture or combination of gestures performed by user 110.); and based on the determined gesture being the first gesture, determine that the interaction is present (para [0041] As another example, one or more of the defined gestures may indicate an interaction by user 110 with a particular user interface element, e.g., selection of UI element 126 of UI menu 124, to trigger a change to the presented user interface, presentation of a sub-menu of the presented user interface, or the like. para [0110] For example, in accordance with the techniques of this disclosure, certain specialized gestures may be pre-defined such that, in response to gesture detector 424 detecting one of the pre-defined gestures, user interface engine 428 dynamically generates a user interface as an overlay to artificial reality content being displayed to the user, thereby allowing the user 110 to easily invoke a user interface for configuring HMD 112 while viewing artificial reality content.).
It would have been obvious to one of ordinary skill in the art before the filing date of present application to have modified the device of Kim and Ibtehaz with the teachings of Ravasz, because this will provide system such that the perform gestures in order to control certain actions while interacting with artificial reality content, thus improving user experience (Ravasz: para [0084]).
Regarding claim 4, Kim and Ibtehaz teaches the electronic device as explained for claim 3 above.
Kim and Ibtehaz fails to teach, wherein the first gesture comprises a pointing gesture for the at least one object; as claimed.
Ravasz teaches the artificial reality system; wherein the first gesture comprises a pointing gesture for the at least one object (para [0089] For example, UI engine 328 may gate the UI element as a menu in portrait orientation if gesture detector 324 detects that the index finger of hand 132 is pointing upward and the back of hand 132 is facing image capture devices 138.).
It would have been obvious to one of ordinary skill in the art before the filing date of present application to have modified the device of Kim and Ibtehaz with the teachings of Ravasz, because this will provide system such that the perform gestures in order to control certain actions while interacting with artificial reality content, thus improving user experience (Ravasz: para [0084]).
Regarding claim 5, Kim and Ibtehaz teaches the electronic device as explained for claim 3 above.
Kim and Ibtehaz fails to teach, wherein the first gesture comprises a pinch gesture for the at least one object
Ravasz teaches the artificial reality system; wherein the first gesture comprises a pinch gesture for the at least one object (para [0100] gesture detector 324 may detect a grip-and-pull combination or a pinch-and-pull combination with respect to the display element that originates at a predefined area of the other arm of user 110, such as at the wrist of the other arm. According to these implementations, UI engine 328 may gate a UI menu of user-selectable options, in response to gesture detector 324 identifying any of these movements. According to some of these implementations, UI engine 328 and rendering engine 322 may change the content, form factor, or selection granularity of the menu in response to gesture detector 324 detecting different lengths of pulling from the other arm's wrist.).
It would have been obvious to one of ordinary skill in the art before the filing date of present application to have modified the device of Kim and Ibtehaz with the teachings of Ravasz, because this will provide system such that the perform gestures in order to control certain actions while interacting with artificial reality content, thus improving user experience (Ravasz: para [0084]).
Regarding claim 14, Kim and Ibtehaz teaches the method as explained for claim 12 above.
Kim and Ibtehaz fails to teach, based on the position of the hand of the user and the joints in the hand, identified using the first AI model, determining a gesture of the hand of the user; identifying whether the determined gesture is a first gesture; and based on the determined gesture being the first gesture, determining that the interaction is present; as claimed.
Ravasz teaches a method, comprising: based on the position of the hand of the user and the joints in the hand, identified using the first AI model (para [0037] o detect the gesture(s), the artificial reality application may compare the motions, configurations, positions and/or orientations of hand 132 and/or portions of arm 134 to gesture definitions stored in a gesture library of artificial reality system 10, where each gesture in the gesture library may be each mapped to one or more actions.), determining a gesture of the hand of the user (para [0108] Similar to the examples described with respect to FIG. 3, based on the sensed data, gesture detector 424 analyzes the tracked motions, configurations, positions, and/or orientations of objects (e.g., hands, arms, wrists, fingers, palms, thumbs) of the user to identify one or more gestures performed by user 110. ); identifying whether the determined gesture is a first gesture (para [0108] Gesture detector 424 may track movement, including changes to position and orientation, of the hand, digits, and/or arm based on the captured image data, and compare motion vectors of the objects to one or more entries in gesture library 430 to detect a gesture or combination of gestures performed by user 110.); and based on the determined gesture being the first gesture, determining that the interaction is present (para [0041] As another example, one or more of the defined gestures may indicate an interaction by user 110 with a particular user interface element, e.g., selection of UI element 126 of UI menu 124, to trigger a change to the presented user interface, presentation of a sub-menu of the presented user interface, or the like. para [0110] For example, in accordance with the techniques of this disclosure, certain specialized gestures may be pre-defined such that, in response to gesture detector 424 detecting one of the pre-defined gestures, user interface engine 428 dynamically generates a user interface as an overlay to artificial reality content being displayed to the user, thereby allowing the user 110 to easily invoke a user interface for configuring HMD 112 while viewing artificial reality content.).
It would have been obvious to one of ordinary skill in the art before the filing date of present application to have modified the method of Kim and Ibtehaz with the teachings of Ravasz, because this will provide system such that the perform gestures in order to control certain actions while interacting with artificial reality content, thus improving user experience (Ravasz: para [0084]).
Regarding claim 15, Kim and Ibtehaz teaches the method as explained for claim 14 above.
Kim and Ibtehaz fails to teach, wherein the first gesture comprises a pointing gesture for the at least one object; as claimed.
Ravasz teaches the method; wherein the first gesture comprises a pointing gesture for the at least one object (para [0089] For example, UI engine 328 may gate the UI element as a menu in portrait orientation if gesture detector 324 detects that the index finger of hand 132 is pointing upward and the back of hand 132 is facing image capture devices 138.).
It would have been obvious to one of ordinary skill in the art before the filing date of present application to have modified the method of Kim and Ibtehaz with the teachings of Ravasz, because this will provide system such that the perform gestures in order to control certain actions while interacting with artificial reality content, thus improving user experience (Ravasz: para [0084]).
Allowable Subject Matter
Claims 2, 6-11 and 13 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: Regarding claim 2, prior art of record fails to teach the following claim limitations of “wherein the first AI model is configured to process image data of a first frame per second (FPS) for the first time period, wherein the second AI model is configured to process image data of a second frame per second (FPS) for the first time period, and wherein, in the first time period, a resource usage of the processor consumed by the second AI model is greater than a resource usage of the processor consumed by the first AI model.”; in combination with all other claim limitations. Regarding claim 6, prior art of record fails to teach the following claim limitations of “…identify whether the at least one object is an object configured to interact; based on the at least one object not being configured to interact, determine that the interaction is absent; based on the at least one object being configured to interact, identify whether a distance between the position of the hand of the user and the at least one object is longer than a first distance; and based on the distance between the position of the hand of the user and the at least one object being longer than the first distance, determine that the interaction is absent.”; in combination with all other claim limitations. Regarding claim 8, prior art of record fails to teach the following claim limitations of “while identifying a position of the hand of the user and a shape of the joints in the hand by using the first AI model or the second AI model to calculate the first data, identify whether the electronic device satisfies a condition; and based on the condition being satisfied, activate a first fixed mode of tracking the hand of the user by fixedly using the first AI model.”; in combination with all other claim limitations. Regarding claim 13, prior art of record fails to teach the following claim limitations of “wherein the first AI model is configured to process image data of a first frame per second (FPS) for the first time period, wherein the second AI model is configured to process image data of a second frame per second (FPS) for the first time period, and wherein, for the first time period, a resource usage of the processor consumed by the second AI model is greater than that of the processor consumed by the first AI model.”; in combination with all other claim limitations.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PREMAL PATEL whose telephone number is (571)270-5892. The examiner can normally be reached Mon-Fri 8-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MATTHEW EASON can be reached at 571-270-7230. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PREMAL R PATEL/Primary Examiner, Art Unit 2624