DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The disclosure is objected to because of the following informalities:
In para. 37, “computer-aided engineering” is accompanied by an incorrect initialism.
The specification recites multiple initialisms. Examples include “RGBD” and “KPIs”. The first instance of an abbreviation, acronym, or initialism should be accompanied by the fully written term.
Appropriate correction is required.
The use of the term “MediaPipe”, which is a trade name or a mark used in commerce, has been noted in this application. The term should be accompanied by the generic terminology; furthermore the term should be capitalized wherever it appears or, where appropriate, include a proper symbol indicating use in commerce such as ™, SM , or ® following the term.
Although the use of trade names and marks used in commerce (i.e., trademarks, service marks, certification marks, and collective marks) are permissible in patent applications, the proprietary nature of the marks should be respected and every effort made to prevent their use in any manner which might adversely affect their validity as commercial marks.
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 14 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 14 recites the limitation "the expert avatar engine" in line 1 of the claim. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without including additional elements that are sufficient to amount to significantly more than the judicial exception itself.
Step 1
The claims are directed to a method and products which fall under the four statutory categories (STEP 1: YES).
Step 2A, Prong 1
Independent claim 1 recites:
A method comprising:
by a computing system:
accessing a posture set from a digital data stream of a target individual performing a task in an environment, wherein postures of the posture set are represented through joint locations of the target individual;
classifying the postures of the posture set into discrete actions;
retrieving target actions from a semantic actions database for performing the task in the environment, wherein the semantic actions database is configured to reference a working context knowledge graph to specify the target actions based on the task and environment conditions of the environment in which the target individual performs the task;
generating guidance for the target individual based on a comparison between the discrete actions classified for the target individual and the target actions retrieved from the semantic actions database; and
providing the guidance to the target individual to assist the target individual in performing the task.
Independent claim 8 recites:
A system comprising:
a semantic actions database configured to reference a working context knowledge graph to specify target actions to perform a task and environment conditions of an environment in which an individual performs the task;
a processor; and
a non-transitory machine-readable medium comprising instructions that, when executed by the processor, cause a computing system to:
access a posture set from a digital data stream of a target individual performing the task in an environment, wherein postures of the posture set are represented through joint locations of the target individual;
classify the postures of the posture set into discrete actions;
retrieve target actions from the semantic actions database for performing the task in the environment;
generate guidance for the target individual based on a comparison between the discrete actions classified for the target individual and the target actions retrieved from the semantic actions database; and
provide the guidance to the target individual to assist the target individual in performing the task.
Independent claim 15:
A non-transitory machine-readable medium comprising instructions that, when executed by a processor, cause a computing system to:
access a posture set from a digital data stream of a target individual performing the task in an environment wherein postures of the posture set are represented through joint locations of the target individual;
classify the postures of the posture set into discrete actions;
retrieve target actions from the semantic actions database for performing the task in the environment;
generate guidance for the target individual based on a comparison between the discrete actions classified for the target individual and the target actions retrieved from the semantic actions database; and
provide the guidance to the target individual to assist the target individual in performing the task.
All of the foregoing underlined elements amount to the abstract idea grouping of a certain method of organizing human activity because it is managing personal behavior or interactions between people (including social activities, teaching, and following rules or instructions) by collecting information (e.g., access a posture set), analyzing it (e.g., classifying, retrieving, generating steps), and outputting the results of the collection and analysis (e.g., providing the guidance). This collection, analysis, and outputting of results also amounts to the abstract idea grouping of mental processes as the claims, under their broadest reasonable interpretation, cover performance of the limitations in the mind with the aid of pen and paper (including observation, evaluation, judgment, opinion) but for the recitation of generic computer components. See MPEP 2106.04(a)(2)(III)(C) - A Claim That Requires a Computer May Still Recite a Mental Process. Even if humans would use a physical aid to help them complete the recited steps, the use of such physical aid does not negate the mental nature of these limitations.
The dependent claims amount to merely further defining the judicial exception.
Therefore, the claims recite a judicial exception. (STEP 2A, PRONG 1: YES).
Step 2A, Prong 2
This judicial exception is not integrated into a practical application because the claims do not include additional elements that are sufficient to integrate the exception into a practical application under the considerations set forth in MPEP 2106.04(d). The elements of the claims above that are not underlined constitute additional elements.
The following additional elements merely generally link the judicial exception to a particular technological environment or field of use: a computing system (claims 1, 8, and 15), a digital data stream (claims 1, 6, 8, 13, 15, and 20), a semantic actions database (claims 1, 8, and 15), a video stream (claims 2, 9, and 16), an augmented reality (AR) device (claims 2, 9, and 16), a virtual reality environment (claims 3, 10, and 17), a system (claim 8), a processor (claims 8 and 15), a non-transitory machine-readable medium (claims 8 and 15), the expert avatar engine (claim 14), . Although the claims recite the elements identified above, these elements are recited at a high level of generality in a conventional arrangement for performing their basic computer functions (i.e., receiving, processing, outputting data). This is evidenced by at least Fig. 1-4 and 6 which illustrate the components as non-descript black boxes or stock images while Fig. 5 recites the process illustrating the claimed invention is focused on the judicial exception itself. Further evidence is provided by the specification. See, for example, at least para. 8-18 of the specification which recite the mere use of augmented reality (AR) and virtual reality (VR) environments to provide guidance and training for any type of requirement, task, or project. See also at least para. 20-26, 36-42, 49-51, 58, 60, and 68-75. Thus, the judicial exception is not implemented with, or used in, a particular machine or manufacture. Additionally, the claims do not recite any limitations that improve the functionality of the computer system. Furthermore, the augmented reality and virtual reality elements that are conventionally configured to perform their conventional functions merely indicates a field of use or technological environment in which to apply a judicial exception. The claims do not apply or use a judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition. For instance, it is silent regarding any specific treatment or prophylaxis for any specific disease or medical condition. Additionally, the additional elements do not apply or use a judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment. In particular, it is merely the use of conventional or generic technology in a nascent but well-known environment (in the instant case, applying existing computer technology), without any assertion that the invention reflects an inventive solution to any problem presented in the technology, itself. Accordingly, based on all of the considered factors, these additional elements do not integrate the abstract idea into a practical application. Therefore, the claims are directed to the judicial exception. (STEP 2A, PRONG 2: NO).
Step 2B
The independent and dependent claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception under the considerations set forth in MPEP 2106.05. As identified in Step 2A, Prong 2, above, the claimed system and the process it performs does not require the use of a particular machine, nor does it result in the transformation of an article. The claims do not involve an improvement in a computer or other technology. Although the claims recite components (identified in Step 2A, Prong 2) for performing at least some of the recited functions, these elements are recited at a high level of generality in a conventional arrangement for performing their basic computer functions (i.e., receiving, processing, outputting data). This is at least evidenced by the manner in which this is disclosed that indicates that the additional elements are sufficiently well-known that the specification does not need to describe the particulars of such additional elements to satisfy 35 USC 112(a) as identified in Step 2A, Prong 2, above. Thus, the judicial exception is not implemented with, or used in, a particular machine or manufacture. Furthermore, this also evidences that the components are merely an attempt to link the abstract idea to a particular technological environment, but do not result in an improvement to the technology or computer functions employed. Additionally, as identified in Step 2A, Prong 2, the mere inclusion of a generically recited AR and VR elements are conventionally configured to perform their conventional functions merely indicates a field of use or technological environment in which to apply a judicial exception. This further evidences that the claims do not recite any specific rules with specific characteristics that improve the functionality of the computer system. Viewed as a whole, these additional claim elements do not provide meaningful limitation to transform the abstract idea into a patent eligible application of the abstract idea such that the claims amount to significantly more than the abstract idea of itself (STEP 2B: NO).
Therefore, the claims are rejected under 35 USC 101 as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-5, 7-12, and 14-19 are rejected under 35 U.S.C. 103 as being unpatentable over Kaleal et al. (US 2022/0384027, hereinafter referred to as Kaleal) in view of Shakeri et al. (US 2016/0189034, hereinafter referred to as Shakeri).
Regarding claims 1, 8, and 15, Kaleal teaches a method (claim 1), a system (claim 8), and a non-transitory machine-readable medium comprising instructions that, when executed by a processor, cause a computing system (claim 15) to:
accessing a posture set from a digital data stream of a target individual performing a task in an environment (Kaleal, para. 61, “client device 106 can include a visual capture device 110 such as a still image camera, a video camera, or a three-dimensional camera/scanner configured to capture image and/or video data of the user 102.”), wherein postures of the posture set are represented through joint locations of the target individual (Kaleal, Fig. 10, Receiving physical information about a user during performance of a physical routine, wherein the physical information includes physiological information and anatomical movement information 1002);
classifying the postures of the posture set into discrete actions (Kaleal, Fig. 10, Receiving physical information about a user during performance of a physical routine, wherein the physical information includes physiological information and anatomical movement information 1002);
retrieving target actions from a semantic actions database for performing the task in the environment, wherein the semantic actions database is configured to specify the target actions based on the task and environment conditions of the environment in which the target individual performs the task (Kaleal, Fig. 10, Determining whether and how the user deviates from requirements of the physical routine based on comparison of the physical information to reference physical metrics for the physical routine 1004);
generating guidance for the target individual based on a comparison between the discrete actions classified for the target individual and the target actions retrieved from the semantic actions database (Kaleal, Fig. 10, Determining a response for performance by an avatar displayed to the user based on and in response to a determination regarding whether and how the user deviates from the requirements of the physical routine, wherein the response is configured to facilitate adherence to the requirements of the physical routine 1006); and
providing the guidance to the target individual to assist the target individual in performing the task (Kaleal, Fig. 10, Initiating manifestation of the response, in response to the determination thereof, by the avatar as displayed to the user during the performance of the physical routine 1008).
Kaleal does not explicitly teach wherein the semantic actions database is configured to reference a working context knowledge graph.
However, Shakeri teaches referencing a working context knowledge graph (para. 56, “a learning (or knowledge) graph may stores data (and metadata) for understanding a person's context (learner, expert, teacher, etc….). The learning graph may also store data and metadata about content materials.”).
Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention for Kaleal to include referencing a working context knowledge graph because it stores data for understanding the user’s context. Thus, it is merely applying a known technique to a known process ready for improvement to yield predictable results.
Regarding claims 2, 9, and 16, Kaleal in view of Shakeri teaches the method of claim 1, the system of claim 8, and the non-transitory machine-readable medium of claim 15, wherein the environment comprises a physical environment (Kaleal, para. 61, “client device 106 can collect video and still images of the user 102 as the user performs an activity, task, or routine (e.g., a workout routine).” Collecting video and still images of the user includes the physical environment the user is in.), and
wherein the posture set is determined from a video stream of the target individual performing the task in the physical environment (Kaleal, para. 61, “The image data can be analyzed using pattern recognition technology to determine whether the user's movement corresponds to model movement metrics for the activity, task or routine.”); and
comprising providing the guidance through an augmented reality (AR) device used by the target individual or another individual in the physical environment (Kaleal, para. 67, “client device 106 can include… heads-up display device (e.g., an augmented reality (AR)… device”).
Regarding claims 3, 10, and 17, Kaleal in view of Shakeri teaches the method of claim 1, the system of claim 8, and the non-transitory machine-readable medium of claim 15, wherein the environment comprises a virtual reality environment and wherein the target individual comprises a user avatar in the virtual reality environment (Kaleal, at least para. 382, “VR devices or headsets correspond to entirely immersive environment wherein the user is presented with a computer-generated simulation of a three-dimensional image or environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment, such as a helmet with a screen inside or gloves fitted with sensors. The field of view in a VR environment does not include a view of the real-world or real-world objects. In some embodiments, 2D and/or 3D avatars can be presented to users in a VR environment (e.g., provided by the avatar platform 2802 and/or other extended reality systems 1722), including role avatars, user identity avatars and animal avatars (as discussed below).”), and
wherein the posture set is determined from the user avatar performing the task in the virtual environment (Kaleal, para. 163, “a representation of the user (e.g., in an avatar form or a video replay of the user) as the user performed the move incorrectly.”); and
comprising providing the guidance through a virtual avatar in the virtual reality environment (Kaleal, at least para. 409, “provided with a visual demonstration (e.g., in AR and/or VR) of an avatar performing how to use the equipment.”).
Regarding claims 4, 11, and 18, Kaleal in view of Shakeri teaches the method of claim 1, the system of claim 8, and the non-transitory machine-readable medium of claim 15, further comprising capturing expert knowledge to store in the semantic actions database, the working context knowledge graph, or a combination of both, including by:
determining a set of actions of an expert individual to perform the task (Kaleal, para. 95, “known requirements for a specific program, routine, or activity can include predetermined reference metrics for the specific program, routine or activity defining values or value ranges for the what the user's physical and physiological activity feature values and/or context parameters should be. In an aspect, these reference metrics 220 are stored in memory 218. These reference metrics can be applied to various algorithms and classification schemes that relate the reference metrics to determined feature values and/or context parameters with respect adherence to of the user to a specific program, routine, or activity.”);
storing the set of actions as the target actions for the task in the semantics actions database (Kaleal, para. 95, “known requirements for a specific program, routine, or activity can include predetermined reference metrics for the specific program, routine or activity defining values or value ranges for the what the user's physical and physiological activity feature values and/or context parameters should be. In an aspect, these reference metrics 220 are stored in memory 218. These reference metrics can be applied to various algorithms and classification schemes that relate the reference metrics to determined feature values and/or context parameters with respect adherence to of the user to a specific program, routine, or activity.”); and
inserting actions of the set of actions, environment conditions for the set of actions, or combinations of both as entries in the working context knowledge graph (Kaleal, para. 95, “known requirements for a specific program, routine, or activity can include predetermined reference metrics for the specific program, routine or activity defining values or value ranges for the what the user's physical and physiological activity feature values and/or context parameters should be. In an aspect, these reference metrics 220 are stored in memory 218. These reference metrics can be applied to various algorithms and classification schemes that relate the reference metrics to determined feature values and/or context parameters with respect adherence to of the user to a specific program, routine, or activity.”).
Regarding claims 7 and 14, Kaleal in view of Shakeri teaches the method of claim 1 and the system of claim 8, further comprising updating the working context knowledge graph or the semantics action database based on analytical processes performed to analyze working context data stored in the working context knowledge graph (Kaleal, para. 388, “All information (e.g., models, image files, audio files, etc.) defining and controlling the appearance and behavior of avatars associated with respective users (e.g., identity avatars, role avatars and animal avatars) can be stored in avatar data 2922 and regularly and/or continuously updated based on new feedback received over time.”).
Regarding claims 5, 12, and 19, Kaleal in view of Shakeri teaches the method of claim 4, the system of claim 11, and the non-transitory machine-readable medium of claim 18.
Kaleal does not explicitly teach wherein determining the set of actions of the expert individual to perform the task comprises exporting an instruction set from an engineering tool.
However these differences are only found in the nonfunctional descriptive material and are not functionally involved in the steps recited. The method would be performed the same regardless of where the instruction set is from. Thus, this descriptive material will not distinguish the claimed invention from the prior art in terms of patentability, see In re Gulack, 703 F.2d 1381, 1385, 217 USPQ 401, 404 (Fed. Cir. 1983); In re Lowry, 32 F.3d 1579, 32 USPQ2d 1031 (Fed. Cir. 1994).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention for determining the set of actions of the expert individual to perform the task in Kaleal to comprise exporting an instruction set from an engineering tool because where the instruction set is exported from is merely a design choice that does not functionally relate to the steps claimed and has not been disclosed to solve any stated problem or is for any particular purpose which does not patentably distinguish the claimed invention.
Claim(s) 6, 13, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kaleal et al. (US 2022/0384027, hereinafter referred to as Kaleal) in view of Shakeri et al. (US 2016/0189034, hereinafter referred to as Shakeri) as applied to claims 4, 11, and 18, further in view of Kumar et al. (US 2006/0073449, hereinafter referred to as Kumar).
Regarding claims 6, 13, and 20, Kaleal in view of Shakeri teaches the method of claim 4, the system of claim 11, and the non-transitory machine-readable medium of claim 18,
Kaleal does not explicitly teach wherein determining the set of actions of the expert to perform the task comprises:
accessing an expert posture set from a digital data stream of the expert individual performing the task, wherein postures of the expert posture set are represented through joint locations of the expert; and
classifying the postures of the expert posture set into discrete actions to form the set of actions of the expert individual.
However, in an analogous art, Kumar teaches wherein determining the set of actions of the expert to perform the task comprises:
accessing an expert posture set from a digital data stream of the expert individual performing the task, wherein postures of the expert posture set are represented through joint locations of the expert (Kumar, para. 24, “expert information 132 containing training results from experts in performing particular training exercises”); and
classifying the postures of the expert posture set into discrete actions to form the set of actions of the expert individual (Kumar, para. 40, “experts may be used to perform the training exercise to provide a ‘gold standard’.”).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention for the reference metrics in Kaleal to be determined by using expert individuals performing the tasks because it provides a gold standard to compare against. See Kumar at para.40.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. All of the prior art disclose using avatars to demonstrate how to perform tasks.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL LANE whose telephone number is (303)297-4311. The examiner can normally be reached Monday - Friday 8:00 - 4:30 MT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xuan Thai can be reached at (571) 272-7147. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL LANE/ Examiner, Art Unit 3715
Prosecution Insights