CTNF 18/794,363 CTNF 82662 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Specification 06-11 AIA The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed. 06-11-01 AIA The following title is suggested: ELECTRONIC DEVICES AND METHODS FOR IDENTIFYING GESTURES ASSOCIATED WITH A TARGET OBJECT TO DETERMINE IF THE GESTURES ARE REAL OR FAKE . Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-21-aia AIA Claim (s) 1, 2, 12, 13 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Al Jazaery (US Pub 2021/0201661) in view of Rao (US Pub 2023/0315209) . Re claim 1: Al Jazaery discloses a method for identifying a gesture associated with a target object, the method comprising: obtaining an image associated with the target object (e.g. the appliance can capture an image associated with a gesture of a user to acquire an image, which is taught in ¶ [30], [31], [39] and [40].); [0030] In some embodiments, when training the image analysis models, gesture image data of the predefined classes of gestures are collected, and a three-dimensional convolutional deep model is trained using the labeled gesture images. Once trained, the convolutional deep module can be used to recognize gestures using input images of users. As disclosed herein, the efficiency of gesture recognition affects the speed by which the gesture is recognized, and the computation power needed to process the images. Using the method and system disclosed herein, the input image for the gesture recognition is very small, resulting in faster recognition without requiring much computational power or a connection to a remote server. This reduces the cost of adding gesture control to an appliance and protects the user's privacy in his home. [0031] As also disclosed herein, utilizing a built-in camera to capture images of a user to control a corresponding appliance is useful. However, sometimes, the user has multiple appliances and multiple appliances may capture the images of the user making the gesture at the same time. Sometimes, not all appliances have the built-in cameras to capture the gesture, even though the user would like to control all appliances with gestures. In this disclosure, the image capturing functions of appliances are optionally shared among multiple appliances (e.g., appliances with cameras and appliances without cameras), and the target appliance for the gesture is not necessarily the appliance that captured the video of the gesture. Carefully designed way to determine a suitable target appliance for a detected gesture is also discussed, such that the gestures are made applicable to more appliances, without requiring all appliances to have a camera and video processing capabilities, and without requiring the user to face a particular appliance or move to a particular location in order to control a desired appliance. [0039] In some embodiments, a respective appliance of the one or more appliances 124 further includes sensors, which senses environment information of the respective appliance. Sensors include but are not limited to one or more light sensors, cameras (also referred to as image sensors), humidity sensors, temperature sensors, motion sensors, weight sensors, spectrometers, and other sensors. In some embodiments, the sensors associated with various appliances are used to provide user presence information (e.g., location of the user in the room, and which appliance(s) that the user is currently interacting with, etc.). In some embodiments, the sensors also provide information on the indoor environment, such as temperature, time of day, lighting, noise level, activity level of the room. This environment information can further be used to select suitable user interface configuration for an appliance, in addition to the recognized gestures of the user that is performed in front of the appliance. [0040] In some embodiments, one or more devices and/or appliances in the kitchen area includes a respective camera and a respective motion sensor to detect presence of a user and captures images of the user. The user can move about the smart kitchen environment, and multiple devices 124 that are located in the vicinity of the user can capture the user's images, and optionally, independently transmit the images to the server system 108 through their own communication channels to the server. In some embodiments the server, optionally, transmits a trained image processing models to one or more of the devices and/or appliances to allow one or more of the devices and/or appliances in the smart home environment to process the images captured in the smart home environment 122 without requiring the images to be transmitted to the server. identifying a region of interest (ROI) within the image, the ROI being associated with the target object (e.g. the invention discloses capturing an input image and identifying a region of interest of a user. The region of interest is associated with the body of the user, which is taught in ¶ [60]-[62] and illustrated in figure 5.); [0060] In some embodiments, during the first stage processing 302, the computing system executes a first image processing process 306 to receive the input image 304 and output one or more regions of interests (ROIs) 308. In some embodiments, the input image 304 is captured by cameras of the appliance (e.g., the cameras 102 of the appliance 124 of FIG. 2). The one or more ROIs 308 correspond to portions of the input image 304 (e.g., portions of the input image 304 that include an upper body of a human user) and the computing system stores the one or more ROIs 308 as new images for further processing (e.g., by the second stage processing 312). [0061] In some embodiments, the first image processing process 306 is a real-time object detection process that identifies the one or more ROIs 308 using machine learning models. For example, the first image processing process 306 can include a You-Only-Look-Once (YOLO) image detection algorithm that utilizes a single convolutional neural network for fast object detection. The first image processing process 306 receives the input image 304 and outputs a vector of bounding boxes and class prediction (e.g., corresponding to the one or more ROIs 308). [0062] In some embodiments, the input image 304 represents a snapshot of a field of view of a camera directed to the physical environment in which the appliance is situated, and the first image processing process 306 is configured to detect in one-pass regions in the input image 304 that include an upper body of a human user. To achieve this, the first image processing process 306 has previously been trained using a first set of training data 307 that includes images labeled with predefined portions of human users (e.g., the upper body of the human user, such as head and shoulder regions of the human user). Therefore, after the computing system executes the first stage processing 302, one or more ROIs including predefined portions of human users (e.g., upper body of human user, including the head and shoulders of the human user) are generated and stored in the computing system for further processing. Refer to FIG. 4 and the related description for detail on how the computing system implements the first stage processing 302. [0065] If the ROI 308 satisfies the further processing conditions 310, the computing system then executes the second stage processing 312 to further process the ROI 308. At the beginning of the second stage processing 312, the ROI 308 is reduced in resolution (e.g., obtaining a smaller size) and stored in the computing system as a reduced ROI 314. A second image processing process 316 then receives the reduced ROI 314 as an input and outputs a candidate control gesture 318. In some embodiments, the candidate control gesture 318 includes a user's hand gesture such as a single-handed hand gesture (e.g., a clenched fist, an open hand, a thumb-up sign, a peace sign, an okay sign, etc.), a two-handed hand gesture (e.g., the Namaste gesture, the Merkel-Raute sign, etc.), or a combination of hand gestures and other body languages. Each candidate control gesture 318 corresponds to a unique digital control command for controlling the appliance. For example, a clenched fist near a user's head may correspond to shutting down the appliance, an open hand may correspond to turning on the appliance, a thumb-up sign may correspond to turning up the power of the appliance, etc. [0066] In some embodiments, the second image processing process 316 includes a real-time single-pass object detection model based on a neural network (e.g., a convolutional neural network) and a classification model (e.g., a support vector machine). The neural network receives the reduced ROIs 314 and determines a corresponding set of intermediate outputs (e.g., a set of predefined features corresponding to the user's hand gestures and head positions), and the classification model then classifies the set of intermediate outputs to a candidate control gesture 318. Each ROI 308 produces a single candidate control gesture 318. In some embodiments, the second image processing process 316 has previously been trained using a second set of training data 315 (e.g., training both the neural network and the classification model). For example, the second set of training data 315 include images corresponding to the size of the reduced ROI 314 with labeled sets of predefined features (e.g., for training the neural network), and mapping between labeled sets of predefined features to candidate control gestures 318 (e.g., for training the classification model). Refer to FIG. 5 and the related description for detail on how the computing system implements the second image processing process 316. PNG media_image1.png 462 670 media_image1.png Greyscale obtaining one or more feature vectors associated with the target object based on the identified ROI (e.g. the first image processing process outputs a vector of features associated with bounding boxes and class prediction corresponding to the identified region of interest, which is taught in ¶ [61], [65] and [66] above.). However, Al Jazaery fails to specifically teach the features of obtaining a traversal path estimate based on the obtained image and the identified ROI, the traversal path estimate being indicative of a region of movement of the target object; identifying, based on the one or more feature vectors and the obtained traversal path estimate, whether the gesture associated with the target object is one of a false gesture or a real gesture. However, this is well known in the art as evidenced by Rao. Similar to the primary reference, Rao discloses identifying gestures of a user (same field of endeavor or reasonably pertinent to the problem). Rao discloses obtaining a traversal path estimate based on the obtained image and the identified ROI, the traversal path estimate being indicative of a region of movement of the target object (e.g. the primary reference discloses the obtaining of an image and an identified region of interest to further evaluate. The Rao invention discloses detecting a hand within a bounding box image. The system further detects the hand movement within the image of the hand present by executing a motion estimation via a motion tracker. The movement of the hand is detected based on the hand signs or gesture sensed by the motion tracker, which is taught in ¶ [56]-[59].); [0056] In an embodiment, the set of sensors 210 may capture an object corresponding to three different hand signs (e.g. a first hand sign 302 , a second hand sign 304 , and a third hand sign 306 ) at three different times T-1, T-2, and T-3. In another embodiment, the set of sensors 210 may capture an object along with the motion of the object, as described, for example, in FIGS. 5 - 8 . In an example, a gesture may include at least two hand signs (e.g. transition from closed palm to open palm). In the example shown in FIG. 3 , the gesture may include a series of three hand signs. The series of the three hand signs may include a transition from closed palm (or “rock” sign) to a closed palm (or “paper” sign) to a V sign (or “scissors” sign). In an embodiment, the neural network model 112 may be optimized based on different techniques to reduce the size of the neural network model 112 . In an embodiment, the neural network model 112 may include an object detector 112 a configured to detect the object and identify the hand sign corresponding to the detected object. In another embodiment, the neural network model 112 may include an image classifier 112 b configured to recognize the hand sign or the gesture and output a class label. In another embodiment, the neural network model 112 may utilize a combination of the object detector 112 a and the image classifier 112 b to recognize the hand sign or the gesture. [0057] In a case where the gesture includes motion of the hand sign, the circuitry 202 may execute a motion tracker (e.g. object tracking or motion estimation using pixel subtraction) on the identified hand sign. The motion tracker may estimate the motion of the object (e.g. hand sign) to determine the gesture based on a look-up table. The details of the motion tracker are further described, for example, in FIGS. 5 - 8 . In order to improve the accuracy of the neural network model 112 for detection of the hand sign, the circuitry 202 may execute a noise suppression technique based on a ring buffer (e.g. the first buffer 108 ), where the frequency of occurrence of each hand sign in a set of consecutive image frames is computed. The number of the set of consecutive images frames may correspond to the size of the ring buffer. The details of the noise suppression technique based on the frequency of occurrence are further described, for example, in FIG. 4 . In another embodiment, the circuitry 202 may execute the noise suppression technique based on a strength of a signal corresponding to each recognized hand sign in the first buffer 108 . The details of the noise suppression technique based on the strength of the signal are further described, for example, in FIG. 9 . [0058] In an embodiment, the object detector 112a may execute object tracking techniques that combine object detection with time-series analysis or object matching/re-detection and tracklet association for object tracking. The object detector 112a may classify the hand sign captured in the image frames, and provide localization information (e.g. location of the detected object across image frames). The object detector 112a may track the movement of the hand sign spatially and temporally. The object detector 112a may determine a bounding box for each object that encloses the detected object in each image frame of the plurality of image frames. The bounding box may corresponds to a set of coordinates in a respective image frame of the plurality of image frames. The object detector 112a may track the bounding box across a threshold number of image frames. The threshold number of image frames may be configurable, and may be set based on different conditions (e.g. image capture conditions, complexity of gesture, etc.). The object detector 112a may estimate the direction of the motion of the object at the end of a predetermined number of tracks or image frames based on the set threshold number of image frames. [0059] In an embodiment, the image classifier 112b may be trained to identify the object based on a set of target classes. For example, the image classifier 112b may be configured to output a class label (e.g. hand sign) for each image frame of the plurality of image frames. The image classifier 112b may be optimized for size constraint of the electronic device 102. For example, the size of the image classifier 112b may be less 100 kilobytes without quantization. The image classifier 112b may have a low inference time of about 50 milliseconds (ms). The circuitry 202 may execute motion estimation based on the output of the image classifier 112b to determine the position of the hand sign and the motion of the hand sign. For example, the circuitry 202 may perform pixel-wise subtraction between consecutive image frames of the plurality of image frames to obtain a motion vector. In an example, the object detector 112a and the image classifier 112b may be built using NNabla® framework. [0060] In an embodiment, the circuitry 202 may utilize a look-up table to interpret the gesture. The look-up table may be configured to store a plurality of gesture classes in association with respective actions. For example, the look-up table may include different combinations of hand signs in association with gesture classes and respective actions. In another example, the look-up table may include different combinations of hand signs and motions in association with gesture classes and respective actions. The circuitry 202 may be configured to combine the output of the neural network model 112 and the output of the motion tracker, and may interpret the meaning of the gesture from the look-up table. In an embodiment, the neural network model 112 , the motion tracker, and the look-up table may be separate entities in the electronic device 102 . For example, the neural network model 112 may output the hand sign as “paper”, and the motion tracker may output the motion of the “paper” hand sign to be clockwise direction from top to bottom. The circuitry 202 may combine the output from the neural network model 112 and the motion tracker, and interpret the gesture as “zoom-in” based on the look-up table. [0061] In another example, the neural network model 112 may output the hand sign as “paper”, and the motion tracker may output the motion of the “paper” hand sign to be anti-clockwise direction from top to bottom. The circuitry 202 may combine the output from the neural network model 112 and the motion tracker, and interpret the gesture as “zoom-out” based on the look-up table. Table 1 shows an example of the look-up table. In Table 1, a gesture may include one of a combination of a hand sign and motion of the hand sign, a combination of a series of hand signs without motion, or a combination of a series of hand signs with motion. It will be understood that the examples of hand signs and motions shown in Table 1 are for description purposes only, and multiple permutations and combinations of hand signs and motions of the hand sings are possible in the look-up table for interpretation of actions. In an embodiment, the look-up table may be customizable and extendable to include any number of gestures and types of gestures and associated actions based on the application of the gesture recognition. [0079] In an embodiment, the neural network model 112 may be an object detector 112 a . In accordance with an embodiment, the circuitry 202 may be configured to apply the object detector 112 a to detect the object 502 , apply the motion tracker 510 to track the motion of the detected object 502 across the plurality of image frames, determine a first gesture 506 based on the tracked motion of the object 502 , and suppress noise in the determination of the first gesture 506 using the first buffer 108 . The circuitry 202 may be configured to set a plurality of quadrants (Q- 1 , Q- 2 , Q- 3 , and Q- 4 ) associated with each image frame of the plurality of image frames. In this case, the granularity of the tracking by the motion tracker 510 may be at a quadrant level. In another embodiment, the circuitry 202 may be configured to set a plurality of grids (e.g. 8×8 grid or 16×16 grid) associated with each image frame of the plurality of image frames. In this case, the granularity of the tracking by the motion tracker 510 may be at a grid level which may be higher than the granularity at the quadrant level. The circuitry 202 may track the motion of the object 502 via the plurality of quadrants or the plurality of grids across the plurality of image frames. As shown in FIG. 5 , the circuitry 202 may set the plurality of quadrants (Q- 1 , Q- 2 , Q- 3 , and Q- 4 ) associated with each of the first set of image frames, the second set of image frames, and the third set of image frames. The circuitry 202 may estimate a direction of the motion of the object 502 (e.g. hand sign) based on the recorded movement of the object 502 from one quadrant to another quadrant. In some embodiments, the circuitry 202 may track the motion of the object 502 across the plurality of grids. [0086] At time T 2 , the circuitry 202 may receive the second image frame of the plurality of consecutive image frames. The circuitry 202 may identify the first hand sign 504 as “rock” in the second image frame based on the output of the image classifier 112 b . The circuitry 202 may be further configured to determine pixel information associated with the second image frame. The motion tracker 510 may be configured to estimate a first motion vector based on first pixel difference information associated with the first image frame and the second image frame. The first pixel difference information may be determined based on a difference between the pixel information associated with the first image frame and the pixel information associated with the second image frame. The motion tracker 510 may be configured to determine a first position of the first hand sign 504 based on the first motion vector. [0087] At time T 3 , the circuitry 202 may receive the third image frame of the plurality of consecutive image frames. The circuitry 202 may identify the first hand sign 504 as “rock” in the third image frame based on the output of the image classifier 112 b . The circuitry 202 may be further configured to determine pixel information associated with the third image frame. The motion tracker 510 may be configured to estimate a second motion vector based on second pixel difference information associated with the second image frame and the third image frame. The second pixel difference information may be determined based on a difference between the pixel information associated with the second image frame and the pixel information associated with the third image frame. The motion tracker 510 may be further configured to determine a second position of the first hand sign 504 based on the second motion vector. [0088] The motion tracker 510 may estimate the direction of the motion of the first hand sign 504 based on the first position and the second position at different time instants. The motion tracker 510 may estimate the direction of the motion of the first hand sign 504 based on movement of the first hand sign 504 across the quadrants. The circuitry 202 may be further configured to determine the first gesture 506 based on the tracked motion of the first hand sign 504 . At the end of time T 3 , the circuitry 202 may determine the first gesture 506 based on the identified first hand sign 504 and the estimated direction of the motion of the first hand sign 504 . The circuitry 202 may determine the first gesture 506 as “zoom-in” based on the look- up table (Table 1). In an embodiment, the circuitry 202 may be configured to display one or more of the movement of the object 502 across the quadrants Q 1 -Q 4 , the first hand sign 504 , and the determined first gesture 506 on a display screen (e.g. the display device 212 ). identifying, based on the one or more feature vectors and the obtained traversal path estimate, whether the gesture associated with the target object is one of a false gesture or a real gesture (e.g. the system identifies a feature that is associated with the motion included within a motion vector. This motion vector used with the estimated motion of the hand can be used to identify the type of gesture the user is performing, which is taught in ¶ [86]-[88] above. The system checks a look-up table to determine the type of gesture being performed, and when the gesture is found, this is considered as a real gesture. This is explained in ¶ [57], [60], [61] above and [127]-[129].). [0127] At 1008, a determination may be made whether a ratio of a number of the first subset of image frames and the first length is one of equal to or greater than a threshold. In at least one embodiment, the circuitry 202 may be configured to determine whether the ratio of the number of the first subset of image frames and the first length is one of equal to or greater than the threshold. The details of the determination of whether the ratio of the number of the first subset of image frames and the first length is one of equal to or greater than the threshold, are described, for example, in FIGS. 3 and 4. [0128] At 1010, storage of the recognized first hand sign in the second buffer 110 may be controlled based on the determination that the ratio is one of equal to or greater than the threshold. In at least one embodiment, the circuitry 202 may be configured to control the storage of the recognized first hand sign in the second buffer 110 based on the determination that the ratio is one of equal to or greater than the threshold. The details of control of the storage of the recognized first hand sign in the second buffer 110 based on the determination that the ratio is one of equal to or greater than the threshold, are described, for example, in FIGS. 3 and 4. [0129] At 1012, a gesture corresponding to one or more hand signs of the plurality of hand signs stored in the second buffer 110 may be determined. In at least one embodiment, the circuitry 202 may be configured to determine the gesture corresponding to one or more of the plurality of hand signs stored in the second buffer 110. The plurality of hand signs may include the first hand sign. The plurality of hand signs may correspond to the second length. The details of determination of the gesture corresponding to one or more of the plurality of hand signs stored in the second buffer 110, are described, for example, in FIGS. 3, 4, 5, 6, 7, and 8. Control may pass to end. Therefore, in view of Rao, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of obtaining a traversal path estimate based on the obtained image and the identified ROI, the traversal path estimate being indicative of a region of movement of the target object; identifying, based on the one or more feature vectors and the obtained traversal path estimate, whether the gesture associated with the target object is one of a false gesture or a real gesture, incorporated in the device of Al Jazaery, in order to determine if a gesture by a user is recognized based on the motion of an image and region of interest, which can enable accurate gesture recognition by optimized network models (as stated in Rao ¶ [22]). Re claim 2: Al Jazaery discloses the method as claimed in claim 1, wherein identifying the ROI associated with the target object comprises: obtaining a plurality of bounding boxes corresponding to the target object (e.g. bounding boxes are generated and obtained that correspond to the parts of the body, which is taught in ¶ [70], [71] and [81].); [0070] The image processing process 400 relies on a deep learning model such as a trained CNN to identify regions of interest including an upper body of a human user. During training of the CNN, training images including various room scenes are labeled to indicate the locations of user's head and shoulders in the training images, and the trained deep learning model is trained to identify the presence of human user's head and shoulders and output their locations in the input images. In some embodiments, training images include images taken with different users in different postures, facing different directions, and at different distances from the camera, and images taken in different times of the day, with different lighting conditions, etc. In some embodiments, the deep learning model is also trained to output the posture of the user (e.g., facing direction of the user), such that only an ROI is only identified when the user in the image is upright and facing the camera (e.g., head and two shoulders are present in the image). In some embodiments, once the user's head location is determined and output by the deep learning model (e.g., the deep learning model is trained to only output the head location when the head is present with two shoulders in the image), the image processing process 400 generates bounding boxes (e.g., bounding boxes 408a-408c) to encompass the identified regions. In some embodiments, the size for each bounding box is determined based on the size of the upper body of the human user in the input image 400. For example, a user closer to the camera (therefore appearing larger in the input image 400) is associated with a larger bounding box (e.g., the bounding box 408a), and a user farther away from the camera (therefore appearing smaller in the input image 400) is associated with a smaller bounding box (e.g., the bounding box 408c). In some embodiments, the bounding box is a box that has a top edge centered at the top of the user's head, and has a width and height that is determined based on the size of the user's head in the image (e.g., the size of the head is generally proportional to the user's arm length and height, and is used as a base unit of length for the size of the bounding box that encloses the region that the user's hands are likely to be found). [0071] Finally, the portions of the input image 402 within the bounding boxes are cropped and normalized to a predefined size (e.g., 400×300 pixel) and stored as output (e.g., the ROIs 308 of FIG. 3) of the image processing pipeline 400 (e.g., cropped images 410). [0081] In some embodiments, the first image processing process is a single-pass detection process (e.g., the first input image is passed through the first image processing process only once and all first ROIs (if any) are identified such as You-Only-Look-Once detection or Single-Shot-Multibox-Detection algorithms). In some embodiments, identifying, using the first image processing process, the one or more first ROIs in the first input image includes: dividing the first input image into a plurality of grid cells (e.g., dividing the first image into an 10×10 grid); for a respective grid cell of the plurality of grid cells: determining, using a first neural network (e.g., the first neural network has previously been trained using labelled images with predefined objects and bounding boxes), a plurality of bounding boxes each encompassing a predicted predefined portion of the human user (e.g., a predicted upper body of the human user, e.g., with the locations of the head and shoulders labeled), wherein a center of the predicted predefined portion of the human user falls within the respective grid cell, and wherein each of the plurality of bounding boxes is associated with a class confidence score indicating a confidence level of a classification (e.g., the type of the object, e.g. a portion of the human body). In some embodiments, the first neural network has previously been trained to detect those classes of objects) of the predicted predefined portion of the human user and a confidence level of a localization of the predicted predefined portion of the human user (e.g., how accurate is the bounding box from the “ground truth box” that surrounds the object. In some embodiments, the class confidence score is a product of localization confidence and classification confidence); and identifying a bounding box with a highest class confidence score in the respective grid cell (e.g., each grid cell will only predict at most one object by removing duplicate bounding boxes through non-maximum suppression process that keeps the bounding box with the highest confidence score and removes any other boxes that overlap the bounding box with the highest confidence score by more than a certain threshold). In some embodiments, the size of the bounding box is selected based on the size of the user's head, and the location of the bounding box is selected based on the location of the user's head identified in the input image. identifying the target object within the image based on the plurality of bounding boxes (e.g. the system identifies the body part within an image based on the bounding boxes around an object, which is taught in ¶ [70], [71] and [81] above.); and identifying the ROI corresponding to the identified target object (e.g. the system identifies the region of interest including hands with the use of the bounding boxes, which is taught in ¶ [82] and [83].). [0082] In some embodiments, the second image processing process is a single-pass object detection process (e.g., You-Only-Look-Once detection or Single-Shot-Multibox-Detection algorithms). In some embodiments, identifying, using the second image processing process, a respective control gesture corresponding to the respective first ROI includes: receiving the downsized copy of the respective first ROI of the plurality of first ROIs; identifying, using a second neural network, a respective set of predefined features of the respective human user; and determining, based on the identified set of predefined features of the respective human user, the respective control gesture. [0083] In some embodiments, the one or more predefined features of the respective human user include one or both hands and a head of the respective human user. In some embodiments, the predefined features include the locations and hand gesture type for each hand identified in the downsized copy of the first ROI. The relative locations of the hand(s) in conjunction with the location of the head (e.g., known from the output of the first image processing process) determines the relative locations of the hand(s) and the head in the first ROI. Re claim 12: Al Jazaery discloses an electronic device for identifying a gesture associated with a target object, the electronic device comprising: a memory configured to store a plurality of modules in the form of programmable instructions; at least one processor, comprising processing circuitry, communicatively coupled to the memory, the at least one processor being configured to, individually and/or collectively, control the system to perform operations comprising: obtaining an image associated with the target object (e.g. the appliance can capture an image associated with a gesture of a user to acquire an image, which is taught in ¶ [30], [31], [39] and [40] above.); identifying a region of interest (ROI) within the image, the ROI being associated with the target object (e.g. the invention discloses capturing an input image and identifying a region of interest of a user. The region of interest is associated with the body of the user, which is taught in ¶ [60]-[62] and illustrated in figure 5 above.); obtaining one or more feature vectors associated with the target object based on the identified ROI (e.g. the first image processing process outputs a vector of features associated with bounding boxes and class prediction corresponding to the identified region of interest, which is taught in ¶ [61], [65] and [66] above.). However, Al Jazaery fails to specifically teach the features of obtaining a traversal path estimate based on the received image and the identified ROI, the traversal path estimate being indicative of a region of movement of the target object; identifying, based on the one or more feature vectors and the obtained traversal path estimate, whether the gesture associated with the target object is one of a false gesture or a real gesture. However, this is well known in the art as evidenced by Rao. Similar to the primary reference, Rao discloses identifying gestures of a user (same field of endeavor or reasonably pertinent to the problem). Rao discloses obtaining a traversal path estimate based on the received image and the identified ROI, the traversal path estimate being indicative of a region of movement of the target object (e.g. the primary reference discloses the obtaining of an image and an identified region of interest to further evaluate. The Rao invention discloses detecting a hand within a bounding box image. The system further detects the hand movement within the image of the hand present by executing a motion estimation via a motion tracker. The movement of the hand is detected based on the hand signs or gesture sensed by the motion tracker, which is taught in ¶ [56]-[59] above.); identifying, based on the one or more feature vectors and the obtained traversal path estimate, whether the gesture associated with the target object is one of a false gesture or a real gesture (e.g. the system identifies a feature that is associated with the motion included within a motion vector. This motion vector used with the estimated motion of the hand can be used to identify the type of gesture the user is performing, which is taught in ¶ [86]-[88] above. The system checks a look-up table to determine the type of gesture being performed, and when the gesture is found, this is considered as a real gesture. This is explained in ¶ [57], [60], [61] and [127]-[129] above.). Therefore, in view of Rao, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of obtaining a traversal path estimate based on the received image and the identified ROI, the traversal path estimate being indicative of a region of movement of the target object; identifying, based on the one or more feature vectors and the obtained traversal path estimate, whether the gesture associated with the target object is one of a false gesture or a real gesture, incorporated in the device of Al Jazaery, in order to determine if a gesture by a user is recognized based on the motion of an image and region of interest, which can enable accurate gesture recognition by optimized network models (as stated in Rao ¶ [22]). Re claim 13: Al Jazaery discloses the electronic device as claimed in claim 12, wherein at least one processor is configured to: obtain a plurality of bounding boxes corresponding to the target object (e.g. bounding boxes are generated and obtained that correspond to the parts of the body, which is taught in ¶ [70], [71] and [81] above.); identify the target object within the image based on the plurality of bounding boxes (e.g. the system identifies the body part within an image based on the bounding boxes around an object, which is taught in ¶ [70], [71] and [81] above.); and identify the ROI corresponding to the detected target object (e.g. the system identifies the region of interest including hands with the use of the bounding boxes, which is taught in ¶ [82] and [83] above.). Re claim 20: Al Jazaery discloses a non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to: obtain an image associated with the target object (e.g. the appliance can capture an image associated with a gesture of a user to acquire an image, which is taught in ¶ [30], [31], [39] and [40] above.); identify a region of interest (ROI) within the image, the ROI being associated with the target object (e.g. the invention discloses capturing an input image and identifying a region of interest of a user. The region of interest is associated with the body of the user, which is taught in ¶ [60]-[62] and illustrated in figure 5 above.); obtain one or more feature vectors associated with the target object based on the identified ROI (e.g. the first image processing process outputs a vector of features associated with bounding boxes and class prediction corresponding to the identified region of interest, which is taught in ¶ [61], [65] and [66] above.); However, Al Jazaery fails to specifically teach the features of obtain a traversal path estimate based on the received image and the identified ROI, the traversal path estimate being indicative of a region of movement of the target object; identify, based on the one or more feature vectors and the obtained traversal path estimate, whether the gesture associated with the target object is one of a false gesture or a real gesture. However, this is well known in the art as evidenced by Rao. Similar to the primary reference, Rao discloses identifying gestures of a user (same field of endeavor or reasonably pertinent to the problem). Rao discloses obtain a traversal path estimate based on the received image and the identified ROI, the traversal path estimate being indicative of a region of movement of the target object (e.g. the primary reference discloses the obtaining of an image and an identified region of interest to further evaluate. The Rao invention discloses detecting a hand within a bounding box image. The system further detects the hand movement within the image of the hand present by executing a motion estimation via a motion tracker. The movement of the hand is detected based on the hand signs or gesture sensed by the motion tracker, which is taught in ¶ [56]-[59] above.); identify, based on the one or more feature vectors and the obtained traversal path estimate, whether the gesture associated with the target object is one of a false gesture or a real gesture (e.g. the system identifies a feature that is associated with the motion included within a motion vector. This motion vector used with the estimated motion of the hand can be used to identify the type of gesture the user is performing, which is taught in ¶ [86]-[88] above. The system checks a look-up table to determine the type of gesture being performed, and when the gesture is found, this is considered as a real gesture. This is explained in ¶ [57], [60], [61] and [127]-[129] above.). Therefore, in view of Rao, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of obtain a traversal path estimate based on the received image and the identified ROI, the traversal path estimate being indicative of a region of movement of the target object; identify, based on the one or more feature vectors and the obtained traversal path estimate, whether the gesture associated with the target object is one of a false gesture or a real gesture, incorporated in the device of Al Jazaery, in order to determine if a gesture by a user is recognized based on the motion of an image and region of interest, which can enable accurate gesture recognition by optimized network models (as stated in Rao ¶ [22]) . 07-22-aia AIA Claim (s) 3 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Al Jazaery, as modified by Rao , as applied to claim s 1 and 12 above, and further in view of Martin (US Pub 2008/0181459) . Re claim 3: However, the combination above fails to specifically teach the features of the method as claimed in claim 1, wherein obtaining the one or more feature vectors associated with the target object comprises: obtaining a plurality of feature points associated with the target object; identifying one or more visible features associated with the target object based on the obtained plurality of feature points; and obtaining the one or more feature vectors associated with the target object based on the one or more visible features. However, this is well known in the art as evidenced by Martin. Similar to the primary reference, Martin discloses detecting hand gestures (same field of endeavor or reasonably pertinent to the problem). Martin discloses wherein obtaining the one or more feature vectors associated with the target object comprises: obtaining a plurality of feature points associated with the target object (e.g. points associated with the hand is obtained, which is taught in ¶ [75]-[79].); [0065] FIG. 1 represents a hand following system HFD according to one embodiment. The system HFD comprises an extraction module for extracting the hand contour HLOC receiving in real time a flow of digital images SV from an acquisition system ACQ such as a video camera, an extraction module for extracting the postural characteristics of the hand FEXT, and a hand movement following module FOLM. The module FEXT extracts the postural characteristics of the hand in each contour image supplied by the module HLOC. The module FOLM determines the hand movements between each image from the postural characteristics extracted by the module FEXT and supplies hand movement information Ml about hand movement between two successive images. [0075] The skeletal pattern of the hand used is represented in FIGS. 4a and 4b. This model comprises markers (marked by circles on FIGS. 4a, 4b) centered on the locations of the center of the palm of the hand PC, joints of the bases of the fingers B1-B5 detected (B1, B2, B3 in FIG. 4b), of the detected fingertips T1-T5 (T1, T2, T3 in FIG. 4b), and of the beginning of the forearm AB. In this skeletal pattern, the marker of the center of the palm of the hand PC is linked by vectors to the marker of the beginning of the forearm AB and to the markers of the bases of the fingers B1-B5 detected. Each finger j (j varying from 1 to 5 in the case in FIG. 4a, and from 1 to 3 in the case in FIG. 4b) comprises a vector linking the marker of its base Bj to the marker of its tip Tj. The positions of the hollows between each finger are also marked on FIGS. 4a, 4b by crosses C1-C6 (C1-C4 in FIG. 4b) each centered on the position of a hollow. [0076] The position of the markers of the fingertips T1-T5 and of the hollows between each finger C1-C6 is determined by calculating the distance between each pixel of the hand contour CT and the center of the palm PC. Thus, FIG. 5 represents a curve 1 of variation of this distance along the hand contour CT represented in FIG. 4a. This curve comprises local minima corresponding to the hollows C1-C6 between each finger, and local maxima corresponding to the fingertips T1-T5. [0077] The bases of the fingers B1-B5 are then located in each image using the position of the markers of the hollows C1-C6. For this purpose, the two hollows Cj, Cj+1 of a finger j are identified by searching for the ones closest to a fingertip Tj considered in the image. The bases of the fingers are then positioned in the middle of the segments [Cj, Cj+1]. However, this mode of calculation is not very accurate for certain fingers, and in particular the thumb one of the hollows of which is poorly located. [0078] Another embodiment of method for locating the base of a finger is shown by FIG. 6. This method comprises selecting out of the two hollows Cj, Cj+1 closest to a fingertip Tj the one closest to the fingertip, i.e., Cj+1 in the example in FIG. 6. This method then comprises searching for the point Pj of the hand contour CT located the closest to the selected hollow Cj+1, and at the same distance from the fingertip Tj as the selected hollow. The base Bj of the finger is then positioned in the middle of the segment linking the selected hollow Cj+1 to the point found Pj. [0079] The module FEXT then determines which finger corresponds to the thumb. For this purpose, the module FEXT applies a test based on the calculation of the angle between the vector of the forearm PC-AB and the vector of each finger Bj-Tj. The thumb is the finger which has the widest angle between these two vectors. Other criteria for determining the thumb can also be applied alternatively or in combination, like the comparison of the lengths of the fingers, the thumb being the shortest finger, and/or of the thicknesses (or widths) of the fingers, the thumb being the thickest (or the widest) finger. The length of each finger j can be determined by calculating the distance Bj-Tj between the markers of the base and of the fingertip. The width of each finger can for example be obtained by calculating the distance between the points of the hand contour CT the closest to a midpoint of the segment Bj-Tj, or an average value of this distance along the segment Bj-Tj. identifying one or more visible features associated with the target object based on the obtained plurality of feature points (e.g. the invention discloses identifying finger tips, the center of a hand and edges of a hand, which is taught in ¶ [75]-[79] above.); and obtaining the one or more feature vectors associated with the target object based on the one or more visible features (e.g. a vector representing movement, or displacement, is based on the movement of points of the hand, which is taught in ¶ [83].). [0083] The module FOLM supplies movement information MI comprising a vector of displacement between two successive images for each marker PC, AB, B1-B5 and T1-T5 of the skeletal pattern. Therefore, in view of Martin, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein obtaining the one or more feature vectors associated with the target object comprises: obtaining a plurality of feature points associated with the target object; identifying one or more visible features associated with the target object based on the obtained plurality of feature points; and obtaining the one or more feature vectors associated with the target object based on the one or more visible features, incorporated in the device of Al Jazaery, as modified by Rao, in order to determine hand movements by using postural characteristics or points extracted from a hand image, which can improve detection of the contours of the hand (as stated in Martin ¶ [08]-[11] and [69]). Re claim 14: However, Al Jazaery fails to specifically teach the features of the electronic device as claimed in claim 12, wherein at least one processor is configured to: obtain a plurality of feature points associated with the target object; identify one or more visible features associated with the target object based on the obtained plurality of feature points; and obtain the one or more feature vectors associated with the target object based on the one or more visible features. However, this is well known in the art as evidenced by Martin. Similar to the primary reference, Martin discloses detecting hand gestures (same field of endeavor or reasonably pertinent to the problem). Martin discloses wherein obtaining the one or more feature vectors associated with the target object comprises: obtain a plurality of feature points associated with the target object (e.g. points associated with the hand is obtained, which is taught in ¶ [75]-[79] above.); identify one or more visible features associated with the target object based on the obtained plurality of feature points (e.g. the invention discloses identifying finger tips, the center of a hand and edges of a hand, which is taught in ¶ [75]-[79] above.); and obtain the one or more feature vectors associated with the target object based on the one or more visible features (e.g. a vector representing movement, or displacement, is based on the movement of points of the hand, which is taught in ¶ [83] above.). Therefore, in view of Martin, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein at least one processor is configured to: obtain a plurality of feature points associated with the target object; identify one or more visible features associated with the target object based on the obtained plurality of feature points; and obtain the one or more feature vectors associated with the target object based on the one or more visible features, incorporated in the device of Al Jazaery, as modified by Rao, in order to determine hand movements by using postural characteristics or points extracted from a hand image, which can improve detection of the contours of the hand (as stated in Martin ¶ [08]-[11] and [69]) . 07-22-aia AIA Claim (s) 4 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Al Jazaery, as modified by Rao and Martin , as applied to claim s 3 and 14 above, and further in view of Common Knowledge of Edge and point detection of an object (Official Notice) . Re claim 4: However, the combination above fails to specifically teach the features of the method as claimed in claim 3, wherein obtaining the plurality of feature points associated with the target object comprises: identifying edges of the target object to generate an edge detected object based on an edge detection model; and obtaining the plurality of feature points from the edge detected object based on a first trained mathematical model. However, this is well known in the art as evidenced by Martin. Similar to the primary reference, Martin discloses detecting hand gestures (same field of endeavor or reasonably pertinent to the problem). Martin discloses wherein obtaining the plurality of feature points associated with the target object comprises: identifying edges of the target object to generate an edge detected object based on an edge detection model (e.g. the system determines the finger tips of the hand or the edges of the hand, which is shown as points in figure 4. The model in ¶ [75] above identifies the edges of the hand.); and obtaining the plurality of feature points from the edge detected object based on a first trained mathematical model (e.g. the system detects the various feature points from the edge of the hand to determine different parts of the hand, which is taught in ¶ [75] above.). [0072] The duration of the processing for locating the hand contour can be reduced by limiting the calculations described previously to a window of the image in which the hand is assumed to be, given the position of the hand in the previous image and/or of the hand movement during the previous images. For this purpose, the module HLOC predicts a window in which the hand is assumed to be. To do so, it implements the standard Kalman model in one embodiment. The standard Kalman model predicts the status of a system from the previous measurements performed on the latter so as to minimize the prediction error covariance matrix. Therefore, in view of Martin, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein obtaining the plurality of feature points associated with the target object comprises: identifying edges of the target object to generate an edge detected object; and obtaining the plurality of feature points from the edge detected object, incorporated in the device of Al Jazaery, as modified by Rao, in order to determine hand movements by using postural characteristics or points extracted from a hand image, which can improve detection of the contours of the hand (as stated in Martin ¶ [08]-[11] and [69]). However, the combination above fails to specifically teach the features of based on an edge detection model; and based on a first trained mathematical model. However, this is well known in the art as evidenced by Common Knowledge of Edge and point detection of an object (Official Notice). Similar to the primary reference, Common Knowledge of Edge and point detection of an object discloses detecting a hand edge and points (same field of endeavor or reasonably pertinent to the problem). Common Knowledge of Edge and point detection of an object discloses based on an edge detection model; and based on a first trained mathematical model (e.g. it is conventional to have a model able to determine the edge of a hand. With the primary reference already able to determine both the edge of the hand and the points on the hand itself, it is well-known to have a model to be able to identify a key point or points within a recognized hand shape to determine or estimate hand position.). Therefore, in view of Common Knowledge of Edge and point detection of an object, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of based on an edge detection model; and based on a first trained mathematical model, incorporated in the device of Al Jazaery, as modified by Rao and Martin, in order to determine outlines and areas of a hand to estimate the position and direction, which can improve the recognition of a hand gesture. Re claim 15: However, Al Jazaery fails to specifically teach the features of the electronic device as claimed in claim 14, wherein at least one processor is configured to: identify edges of the target object to generate an edge detected object based on an edge detection model; and obtain the plurality of feature points from the edge detected object based on a first trained mathematical model. However, this is well known in the art as evidenced by Martin. Similar to the primary reference, Martin discloses detecting hand gestures (same field of endeavor or reasonably pertinent to the problem). Martin discloses wherein at least one processor is configured to: identify edges of the target object to generate an edge detected object (e.g. the system determines the finger tips of the hand or the edges of the hand, which is shown as points in figure 4. The model in ¶ [75] above identifies the edges of the hand.); and obtaining the plurality of feature points from the edge detected object (e.g. the system detects the various feature points from the edge of the hand to determine different parts of the hand, which is taught in ¶ [75] above.). Therefore, in view of Martin, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of wherein at least one processor is configured to: identify edges of the target object to generate an edge detected object; and obtaining the plurality of feature points from the edge detected object, incorporated in the device of Al Jazaery, as modified by Rao, in order to determine hand movements by using postural characteristics or points extracted from a hand image, which can improve detection of the contours of the hand (as stated in Martin ¶ [08]-[11] and [69]). However, the combination above fails to specifically teach the features of based on an edge detection model; and based on a first trained mathematical model. However, this is well known in the art as evidenced by Common Knowledge of Edge and point detection of an object (Official Notice). Similar to the primary reference, Common Knowledge of Edge and point detection of an object discloses detecting a hand edge and points (same field of endeavor or reasonably pertinent to the problem). Common Knowledge of Edge and point detection of an object discloses based on an edge detection model; and based on a first trained mathematical model (e.g. it is conventional to have a model able to determine the edge of a hand. With the primary reference already able to determine both the edge of the hand and the points on the hand itself, it is well-known to have a model to be able to identify a key point or points within a recognized hand shape to determine or estimate hand position.). Therefore, in view of Common Knowledge of Edge and point detection of an object, it would have been obvious to one of ordinary skill before the effective filing date of the claimed invention was made to have the feature of based on an edge detection model; and based on a first trained mathematical model, incorporated in the device of Al Jazaery, as modified by Rao and Martin, in order to determine outlines and areas of a hand to estimate the position and direction, which can improve the recognition of a hand gesture . Allowable Subject Matter 12-151-08 AIA 07-43 12-51-08 Claim s 5-11 and 16-19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. 13-03-01 AIA The following is a statement of reasons for the indication of allowable subject matter: The subject matter associated with claims 5, 8 and 10 were not found in the applied and/or cited prior art . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Mazurenko discloses a grid over a hand. Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHAD S DICKERSON whose telephone number is (571)270-1351. The examiner can normally be reached Monday-Friday 10AM-6PM EST.. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Abderrahim Merouan can be reached at 571-270-5254 . The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /CHAD DICKERSON/ Primary Examiner, Art Unit 2683 Application/Control Number: 18/794,363 Page 2 Art Unit: 2683 Application/Control Number: 18/794,363 Page 3 Art Unit: 2683 Application/Control Number: 18/794,363 Page 4 Art Unit: 2683 Application/Control Number: 18/794,363 Page 5 Art Unit: 2683 Application/Control Number: 18/794,363 Page 6 Art Unit: 2683 Application/Control Number: 18/794,363 Page 7 Art Unit: 2683 Application/Control Number: 18/794,363 Page 8 Art Unit: 2683 Application/Control Number: 18/794,363 Page 9 Art Unit: 2683 Application/Control Number: 18/794,363 Page 10 Art Unit: 2683 Application/Control Number: 18/794,363 Page 11 Art Unit: 2683 Application/Control Number: 18/794,363 Page 12 Art Unit: 2683 Application/Control Number: 18/794,363 Page 13 Art Unit: 2683 Application/Control Number: 18/794,363 Page 14 Art Unit: 2683 Application/Control Number: 18/794,363 Page 15 Art Unit: 2683 Application/Control Number: 18/794,363 Page 16 Art Unit: 2683 Application/Control Number: 18/794,363 Page 17 Art Unit: 2683 Application/Control Number: 18/794,363 Page 18 Art Unit: 2683 Application/Control Number: 18/794,363 Page 19 Art Unit: 2683 Application/Control Number: 18/794,363 Page 20 Art Unit: 2683 Application/Control Number: 18/794,363 Page 21 Art Unit: 2683 Application/Control Number: 18/794,363 Page 22 Art Unit: 2683 Application/Control Number: 18/794,363 Page 23 Art Unit: 2683 Application/Control Number: 18/794,363 Page 24 Art Unit: 2683 Application/Control Number: 18/794,363 Page 25 Art Unit: 2683 Application/Control Number: 18/794,363 Page 26 Art Unit: 2683 Application/Control Number: 18/794,363 Page 27 Art Unit: 2683 Application/Control Number: 18/794,363 Page 28 Art Unit: 2683 Application/Control Number: 18/794,363 Page 29 Art Unit: 2683 Application/Control Number: 18/794,363 Page 30 Art Unit: 2683
Prosecution Insights