CTNF 18/890,493 CTNF 86455 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim s 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over D1 1 and further in view of D2. 2 With regard to claim 1 , D1 teach a client device comprising a display, a memory storing instructions; and one or more processors operable to execute the instructions, stored in the memory ( see fig. 4, § 3.3 ¶¶ 1-3: smart glasses comprising a display, memory and processor ), to: receive, at the client device, user input that visually indicates a gesture of a user and a personal identity of the user ( see fig. 4, § 3.3 ¶¶ 1-3: smart glasses capture images of a person performing a gesture including, the picture includes image of the face ); generate, based on processing the user input using a machine learning model, anonymized data that indicates the gesture of the user and that anonymizes the personal identity of the user ( see fig. 5, § 3.3 ¶¶ 1-4: MediaPipe is used to generate gesture data, MediaPipe inherently uses machine learning models ); transmit a subset of the anonymized data to a computing device ( see fig. 5, abstract, § 3.3 ¶ 4, § 1 ¶ 6: transmits data to cloud based system that translates the sign language ); receive, at the client device and from the computing device, natural language interpretation data that corresponds to the anonymized data, and that identifies a natural language interpretation of the gesture of the user ( see fig. 5, § 3.3 ¶ 4: translation result is transmitted back to the smart glasses ); and perform an action based on the natural language interpretation of the gesture of the user ( see figs. 4-5: the translation is presented to the wearer of the smart glasses who may then perform an action based on the translation; see also § 1 ¶ 4: gestures interpreted and actions such as drawing, writing and deletion operations are performed ). D1 uses MediaPipe framework to recognize and collect sign language action data from sign language videos by detecting hand movement and mapping the hand feature points and the detected feature data is sent to the a signa language translation server for translating the sign language ( see D1 § 1 ¶ 6, § 4.2 ¶ 3 ). By mapping the hand features, MediaPipe implicitly suggests that at least some of the data is anonymized before sending it to the server, however D1 does not explicitly teach anonymization of data. However, D2 teaches the missing features ( see D2 abstract, § IIA: anonymizing data by excluding data corresponding to the face ). One skilled in the art before the effective filing date would have found it obvious to combine the teachings to arrive at the claimed invention. In particular, it would have been obvious to incorporate known teachings of D2 which uses MediaPipe in to the configuration of D1 yielding predictable results. In particular, it would have been obvious to incorporate the teachings of anonymizing the data into D2 such that only a subset of data is transmitted to the server for translation. The motivation would have been to anonymize the data to protect the privacy of the user before transmitting it to the cloud. With regard to claim 2 , D2 teach wherein the user input includes both visual input and audio input, and the anonymized data that is transmitted to the computing device only includes visual input ( see fig. 4, § 3.1 ¶ 1: smart glasses capture video which implicitly includes audio data, MediaPipe processes the video data but only transmits gesture data to the cloud for translating the sign language ). With regard to claim 3 , D1 teach wherein the gesture of the user is a non-verbal communicative input ( see fig. 4: hand gesture data captured by the smart glasses ). With regard to claim 4 , D2 teach identify a face, of the user, in the user input ( see fig. 1, § II ¶ 1, § IIA, § IIB : face is identified, including the nose ); process characteristics of the face ( see fig. 1, § II ¶ 1, § IIA, § IIB : face features are processed, nose identified ); and generate, based on processing the characteristics of the face, an anonymized point mapping of the characteristics of the face, wherein the anonymized data is generated based on the anonymized point mapping of the characteristics of the face, and wherein the subset of the anonymized data includes a subset of the anonymized point mapping of the characteristics of the face ( see fig. 1, § IIB ¶¶ 1-4, fig. 3: face is anonymized by mapping the feature points to a cartoon character, subset of anonymized data is generated by excluding some features of the face, see § IIA ). The motivation for combining the references is the same as stated above. With regard to claim 5 , D2 teach wherein the characteristics of the face include one or more of: an eye, eyebrow, mouth, cheek, nose, and ear ( see fig. 1, § IIA: nose ). With regard to claim 6 , D2 teach normalize, using a default proportional template, the anonymized point mapping of the characteristics of the face, wherein proportions of the characteristics of the face are different from proportions of the default proportional template, and wherein the anonymized data is generated based on normalizing the anonymized point mapping of the characteristics of the face ( see fig. 1, § II ¶ 1, § IIA, § IIB: anonymized point mapping of the features to a cartoon template with different proportions to generate normalized data ). The motivation for combining the references is the same as stated above. With regard to claim 7 , D2 teach identify body segments of the user, in addition to the face of the user, in the user input ( see fig. 1, § II ¶ 1: body segments including face, upper body, hands ); process characteristics of the body segments of the user, in addition to processing characteristics of the face of the user; and generate, based on processing the characteristics of the body segments, an anonymized point mapping of the characteristics of the body segments, wherein the anonymized data is generated based on the anonymized point mapping of the characteristics of the body segments ( see fig. 1, §§ IIA, IIB ¶¶ 1-4: face, torso and hands are processed and mapping to cartoon skeleton template to generate anonymized data ). The motivation for combining the references is the same as stated above. With regard to claim 8 , D2 teach wherein the body segments are above a waist of the user and include one or more of a hand of the user and a torso of the user ( see fig. 1, §§ IIA, IIB ¶¶ 1-4: body segments include torso, hands ). With regard to claim 9 , D2 teach wherein the machine learning model is a point mapping model ( see abstract, fig. 1, §§ IIA, IIB ¶¶ 1-4: machine learning based retargeting or mapping points of the person to corresponding points on a cartoon skeleton template ). With regard to claim 10 , D2 teach wherein the machine learning model is a media pipe holistic model ( see § IIA: MediaPipe ). With regard to claim 11 , D1 teach wherein the gesture of the user includes an American Sign Language (ASL) communicative input ( see abstract, § I ¶ 6: WLASL (World Level American Sign Language )). With regard to claim 12 , D1 teach wherein the natural language interpretation data corresponds to an interpretation of the ASL communicative input ( see abstract: natural language interpretation or translation of the sign language ). With regard to claim 13 , D1 fails to teach wherein the user input captures a first portion of the user and a second portion of the user, and wherein the instructions to process the user input using the machine learning model include instructions to: process the first portion of the user at a first framerate; and process the second portion of the user at a second framerate. However, Examiner takes Official Notice to the fact that it is extremely well known in the art before the effective filing date to capture images at different frame rates. It would have been particularly obvious to incorporate known teachings of capturing at different frame rates into the configuration of D1 yielding predictable and enhanced results. The motivation for using different frame rates would have been to reduce required computational resources by capturing at a lower frame rate for example. Because hands and face have more motion compared to torso when performing gestures, it would have been obvious to capture the hands and face at a higher frame rate and use a lower frame rate to capture the torso which does not involve as much motion when performing sign language gestures. With regard to claim 14 , D1 and D2 teach wherein the memory further comprises instruction to, prior to executing an instruction to perform the action based on the natural language interpretation of the gesture of the user: receive, at the client device and during an interaction that the gesture is received, further user input that visually identifies the personal identity of the user and that identifies another gesture of the user that is in furtherance of the interaction; generate, based on processing the further user input using the machine learning model, additional anonymized data that indicates the other gesture of the user and anonymizes the personal identity of the user; transmit an additional subset of the additional anonymized data to the computing device; receive, at the client device and from the computing device, additional natural language interpretation data that corresponds to the additional anonymized data, and that identifies a natural language interpretation of the other gesture of the user; and determine, based on processing the natural language interpretation of the gesture of the user and the natural language interpretation of the other gesture of the user, the action to be performed. See discussion of claims above. Claim 14 merely repeats processing for additional sign language video, which can be processed the same way as discussed in the claims above. With regard to claim 15 , D1 does not teach wherein the memory further comprises instructions to, prior to executing an instruction to generate the anonymized data: determine a failure to locally generate, based on processing the user input using the machine learning model or another machine learning model locally at the client device, the natural language interpretation of the gesture of the user or another natural language interpretation of the gesture of the user, wherein the instructions to generate the anonymized data are executed in response to determining the failure to locally generate the natural language interpretation of the gesture of the user or the other natural language interpretation of the gesture of the user. However, it would have been obvious for one skilled in the art to generate anonymized data only when transmitting the data to a server or a cloud where private information may be exploited. Therefore, it would have been obvious to generate anonymized data only when it is necessary to transmit to the cloud or server, which would be necessary when local computational resources are insufficient for processing. Alternatively, Examiner takes Official Notice to the fact that it is well known in the art before the effective filing date to check the processing capacity of the local device to ascertain if it is sufficient to perform processing task, and if the determination is that it is insufficient, then to transmit the task to a server or cloud for processing. It would have been obvious to incorporate known teachings into the configuration of D1 yielding predictable results. With regard to claim 16 , D1 teach wherein the natural language interpretation of the gesture of the user indicates an English natural language interpretation of American Sign Language communicative input that is included in the user input ( see abstract: natural language interpretation or translation of the sign language implicit that it is English, see abstract, § I ¶ 6: WLASL (World Level American Sign Language )). With regard to claim 17 , D1 fails to teach wherein the gesture of the user is a request for a search query to be performed, and performing the action includes performing at least one of the search query or another search query associated with the search query. However, this merely appears to be an intended use limitation. The teachings of D1 may be used to translate a gesture requesting a search query to be performed and then to perform that action. With regard to claim 18 , see discussion of claim 1. With regard to claim 19 , D1 teach wherein the data that is indicative of the gesture of the user is indicative of one or more of a hand gesture of the user and a facial gesture of the user ( see fig. 4: hand gesture ). With regard to claim 20 , see discussion of claim 1. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to AVINASH YENTRAPATI whose telephone number is (571)270-7982. The examiner can normally be reached on 8AM-5PM. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice . If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached on (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov . Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /AVINASH YENTRAPATI/Primary Examiner, Art Unit 2672 Application/Control Number: 18/890,493 Page 2 Art Unit: 2672 Application/Control Number: 18/890,493 Page 3 Art Unit: 2672 Application/Control Number: 18/890,493 Page 4 Art Unit: 2672 Application/Control Number: 18/890,493 Page 5 Art Unit: 2672 Application/Control Number: 18/890,493 Page 6 Art Unit: 2672 Application/Control Number: 18/890,493 Page 7 Art Unit: 2672 Application/Control Number: 18/890,493 Page 8 Art Unit: 2672 Application/Control Number: 18/890,493 Page 9 Art Unit: 2672 1 Zhao, Siwei, et al. "A cloud-based sign language translation system via CNN with smart glasses." International Conference on Green, Pervasive, and Cloud Computing. Singapore: Springer Nature Singapore, 2023. 2 Tze, Christina O., et al. "Cartoonized anonymization of sign language videos." 2022 IEEE 14th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP). IEEE, 2022.
Prosecution Insights