Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This Final Office action is responsive to the communication filed under 37 C.F.R. § 1.111 on October 27, 2025 (hereafter “Response”). The amendments to the claims are acknowledged and have been entered.
Claims 1 and 11 are now amended.
Claims 1–14 are pending in the application.
Response to Arguments
The objections to the drawings, specification, and claims are hereby withdrawn, responsive to the Applicant’s corrections thereof.
Claim(s) 1–14 stand rejected under 35 U.S.C. § 103 as being unpatentable over U.S. Patent Application Publication No. 2008/0235788 A1 (“El Saddik”) in view of Erhu Zhang et al., Fusion of 2D CNN and 3D DenseNet for Dynamic Gesture Recognition, 8 Electronics 12, 1511 (Dec. 9, 2019), https://doi.org/10.3390/electronics8121511 (“Zhang”). The Applicant’s arguments have been considered in light of the amendment, but are not persuasive of nonobviousness.
The Applicant’s arguments are not persuasive because they attempt to read the references in isolation of one another, when the rejection is based on what the combination of those references taught and suggested to those of ordinary skill in the art before the effective filing date of the claimed invention. El Saddik indeed provides tuples to its ANN rather than a finished “image,” but Zhang teaches the same technique as claimed, i.e., first generating an image from the input data, in order to leverage the power and accuracy of convolutional neural networks, which are configured to recognize images, rather than input data per se. The present rejection is based on the obviousness of applying Zhang’s technique to El Saddik’s tuples, thus converting them into images (e.g., using the known rainbow encoding technique for position and extent data).
The Applicant’s discussion of Zhang (Response 9) makes the same mistake. The Applicant argues that Zhang “does not require taking positional and extent data from a sensing array and mapping this to a corresponding pixel array,” but again, the rejection is made in view of the obviousness of applying Zhang’s technique to El Saddik, which collects positional and extent data from a sensing array.
Accordingly, since the Applicant’s remarks address the references separately rather than their combination, the Examiner is not persuaded of error, and the rejection stands.
Claim Rejections – 35 U.S.C. § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1–14 are rejected under 35 U.S.C. § 103 as being unpatentable over U.S. Patent Application Publication No. 2008/0235788 A1 (“El Saddik”) in view of Erhu Zhang et al., Fusion of 2D CNN and 3D DenseNet for Dynamic Gesture Recognition, 8 Electronics 12, 1511 (Dec. 9, 2019), https://doi.org/10.3390/electronics8121511 (“Zhang”).
Claim 1
El Saddik teaches:
A method of classifying a mechanical interaction on a sensing array, said sensing array comprising a plurality of sensing elements, said method comprising the steps of:
“FIG. 6 shows a method of passgraph entry,” El Saddik ¶ 58, using a “haptic entry input device” that is capable of “generating kinestatic and/or tactile output in either single-point or multi-point interaction.” El Saddik ¶ 56. Specifically an “entry grid may be placed on for example an elastic membrane of a touch pad 513 providing force feedback resistance and friction when the pen's end-effecter or users finger makes contact with the virtual grid object.” El Saddik ¶ 56.
identifying positional and extent data in response to said mechanical interaction on said sensing array;
“The user commences passgraph entry at step 602 by the system determining if one of the grid points has been selected, if the grid point has been selected, the haptic state of the input device is determined at that point at step 604.” El Saddick ¶ 58.
mapping said positional and extent data from said plurality of sensing elements in said sensing array to a plurality of pixels in a pixel array, each of said plurality of pixels corresponding to one of said plurality of sensing elements;
The customary meaning of “pixel” refers to the smallest element manipulable by a given piece of display hardware, amongst a grid of such elements. See Microsoft Press, Microsoft Computer Dictionary 406 (5th Ed. 2002) and IEEE, The Authoritative Dictionary of IEEE Standards Terms 827 (7th ed. 2000). The present application does not provide any special overriding definition of “pixel,” so its customary meaning prevails as the broadest reasonable interpretation.
El Saddick likewise teaches that its grid 100 for entering passgraphs “is defined by an x axis 102 and y axis 104” consisting of a predetermined number of respective columns and rows, and that, for every “intersection of rows and columns[,] an entry point 106 is defined.” El Saddick ¶ 29. Hence, these rows and columns of entry points 106 fall within the scope of the claimed pixel array.
With respect to “mapping said positional and extent data from said plurality of sensing elements in said sensing array to a plurality of pixels in a pixel array,” El Saddick further teaches that for each entry point 106 selected, “[a] tuple containing the coordinates of the assigned grid system of the passgraph is generated at step 606,” and then steps 602–606 are repeated for each subsequent portion of the input. El Saddick ¶ 58.
converting said positional and extent data to image data to produce an image recorded over a series of frames,
“If no more tuples are required, NO at step 608 the password is generated at step 610 by combining tuples.” El Saddick ¶ 58. “FIG. 2 shows an example password that the user may enter into the grid. The graphical password shown is composed of two separate lines 202 and 206. The bold lines 204, which is a portion of 202, and a second bold line 206 indicate places where the user has put more pressure in creating the password. The term passgraph is used to describe a graphical representation of the user's password.” El Saddick ¶ 30.
said image comprising a three-layer image, and each layer corresponding to a different
The tuples combined at step 610 are each defined as “(x, y, p), where x corresponds to the position along the x-axis of the passgraph, y corresponds to the position along the y-axis of the passgraph, and p is the value determined for the haptic input state.” El Saddick ¶ 58.
classifying said positional and extent data by providing said image to an artificial neural network.
“Once the password has been generated it can then be verified 612 against the stored password.” El Saddick ¶ 58. “The identification methods of the complete entry can be performed using a number of pattern recognition techniques such as artificial neural network (ANN).” El Saddick ¶ 48.
Despite using haptic intensity to define stroke intensity in the passgraph, El Saddick does not explicitly disclose that the passgraph is a color image, let alone one where color is dependent upon time.
Zhang, however, teaches another method, but improved by a rainbow-encoding technique, comprising:
identifying positional and extent data
As shown in Figure 1, a camera produces input data of an RGB video of a gesture performed in front of the camera. Zhang 4.
converting said positional and extent data to image data to produce an image recorded over a series of frames, said image comprising a three-layer image, and each layer corresponding to a different color output measured over a period of time; and
Still referring to Figure 1, the RGB videos are converted into a “pseudo-color image.” Zhang 4. To produce the pseudo-color image, first the video is mixed down into a single, static “motion history image” that “is used to encode the motion information of a sequence of gesture videos into a single image” denoted as Hτ (x, y, t), “where (x, y) is the pixel position and t is the frame number.” Zhao 5. Then, “to further enhance the spatiotemporal information contained in the MHI gray image, the MHI is transformed into a pseudo-colored image by the rainbow encoding method.” Zhang 7.
The rainbow encoding method is shown in Equation 5. In short, it converts the gray image Hτ to an RGB image with the three claimed “layers,” one for each channel of RGB: (Cr, Cg, Cb). Zhang 7. Figure 5 illustrates the resulting image with greater detail, although its color will not be visible in the file wrapper. Those reviewing this rejection are urged to obtain the original copy of Zhang’s paper from its source at <https://doi.org/10.3390/electronics8121511>. A direct link to Figure 5 is also available at <https://www.mdpi.com/electronics/electronics-08-01511/article_deploy/html/images/electronics-08-01511-g005.png>
classifying said positional and extent data by providing said image to an artificial neural network.
The pseudo-color image is input into CNN that is based on VGGNet, having a third fully connected layer where “the number of its nodes is equal to the number of gesture classes,” and concluding with a Softmax classifier to output the final decision about the gesture. Zhang 8.
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to improve El Saddik’s method for recognizing gestures with Zhang’s known technique of pseudo-coloring the input gesture to provide additional spatiotemporal information about the gesture. One would have been motivated to improve El Saddik’s method with the pseudo-coloring technique because pseudo-coloring a motion history image “further increase[s] the distinctiveness of motion representation.” Zhang 3.
Claim 2
El Saddik and Zhang teach the method of claim 1,
wherein said step of identifying positional and extent data identifies two-dimensional location data and a magnitude of force applied.
The tuples combined at step 610 are each defined as “(x, y, p), where x corresponds to the position along the x-axis of the passgraph, y corresponds to the position along the y-axis of the passgraph, and p is the value determined for the haptic input state.” El Saddick ¶ 58.
Claim 3
El Saddik and Zhang teach the method of claim 1,
wherein said extent data is defined as a range of levels corresponding to a numerically similar range of levels of each said color output.
“[A] multilevel input may be provided where p is mapped to defined pressure levels, for example three levels of pressure.” El Saddick ¶ 33. El Saddick is combined with Zhang, which colorizes the image according to its motion history, and thus the result is a colorized image in which intensity levels are defined by the p dimension representing pressure at each point in time.
Claim 4
El Saddik and Zhang teach the method of claim 1,
wherein a value of said extent data is presented in said image by a brightness of said color output.
The tuples combined at step 610 are each defined as “(x, y, p), where x corresponds to the position along the x-axis of the passgraph, y corresponds to the position along the y-axis of the passgraph, and p is the value determined for the haptic input state.” El Saddick ¶ 58. El Saddick is combined with Zhang, which colorizes the image according to its motion history, and thus the result is a colorized image in which intensity levels are defined by the p dimension representing pressure at each point in time.
Claim 5
El Saddik and Zhang teach the method of claim 1,
wherein said artificial neural network is a convolutional neural network.
The pseudo-color image is input into CNN that is based on VGGNet, having a third fully connected layer where “the number of its nodes is equal to the number of gesture classes,” and concluding with a Softmax classifier to output the final decision about the gesture. Zhang 8.
Claim 6
El Saddik and Zhang teach the method of claim 5, further comprising the step of:
pre-training said convolutional neural network to interpret said image as a predetermined gesture.
“Once the motion representation is computed from gesture videos, the next stage is to build a 2D CNN model for extracting the features from this motion representation.” Zhang 8.
Claim 7
El Saddik and Zhang teach the method of claim 1, further comprising the step of:
providing a repeated mechanical interaction to said artificial neural network to establish a classification image.
“The ANN can be trained using the back propagation unsupervised learning technique for 5000 epochs (steps in the training process).” El Saddik ¶ 49.
Claim 8
El Saddik and Zhang teach the method of claim 1, further comprising the step of:
removing background noise by means of said artificial neural network.
“To make MHI less sensitive to light change and silhouette noise, the RGB image sequence is firstly converted to grayscale images,” Zhang 6, and then the converted image is re-colorized at the input stage of the 2D CNN.
Claim 9
El Saddik and Zhang teach the method of claim 1, further comprising the step of:
confirming said classification step by providing an output in response to said mechanical interaction.
“Comparisons between the sample profile and the templates associated with the claimed identity produce a quantitative verification Match Score (MS).” El Saddik ¶ 51.
Claim 10
El Saddik and Zhang teach:
A touch screen comprising said sensing array and a processor configured to perform the method of claim 1.
“FIG. 5 shows a computer system for implementing a graphical password entry system,” which includes a “haptic entry input device 511” as well as a “CPU 504,” El Saddik ¶ 56, that is configured via program instructions (e.g., stored in one or more of 506, 518, 520, 522, and 524) to perform the method discussed earlier. See El Saddik ¶ 57.
Claim 11
Claim 11 is rejected over the same findings and rationale as provided above for claim 10 (which includes the findings and rationale provided for the rejection of its parent claim 1).
Claim 12
El Saddik teaches the apparatus of claim 11,
wherein said sensing array comprises a first plurality of sensing elements arranged in rows and a second plurality of sensing elements arranged in columns;
“In FIG. 1, the grid 100, is defined by an x axis 102 and y axis 104 each having 8 columns and rows respectively.” El Saddik ¶ 29. “The haptic entry input device may any device cable of generating kinestatic and/or tactile output in either single-point or multi-point interaction device. The device may for example comprise a pen 512, a touchpad 513, a touch screen 514, single-point interaction devices 515 such as a pressure sensor or button, or multi-point devices 516 such as hand or body sensing technologies.” El Saddik ¶ 56.
and said image comprises a corresponding first plurality of pixels arranged in rows and a corresponding second plurality of pixels arranged in columns.
“The information captured from a password drawn in FIG. 2 is mapped to a tuple (x,y,p), where x and y represents the position of the selected points on the horizontal and vertical axis respectively and p is a binary input indicating if high (more than the user's average) pressure is exerted when two points on the grid are connected. The tuple (-1,-1,-1) is recorded when a pen-up occurs. For example, the data recorded from FIG. 2 are listed as follows: (1,6,0), (2,6,0), (3,6,0), (4,6,0), (4,5,0), (4,4,0), (5,4,1), (6,4,1), (7,4,1), (7,5,0), (7,6,0), (7,7,0), (7,8,0), (-1,-1,-1) (6,6,0), (6,7,1), (6,8,1) (-1,-1,-1).” El Saddik ¶ 31.
Claim 13
El Saddik and Zhang teach the apparatus of claim 12,
wherein a value of said extent data is presented in said image by a brightness of said color output.
“[A] multilevel input may be provided where p is mapped to defined pressure levels, for example three levels of pressure.” El Saddick ¶ 33. El Saddick is combined with Zhang, which colorizes the image according to its motion history, and thus the result is a colorized image in which intensity levels are defined by the p dimension representing pressure at each point in time.
Claim 14
El Saddik and Zhang teach the apparatus of claim 11,
wherein said artificial neural network is a convolutional neural network.
The pseudo-color image is input into CNN that is based on VGGNet, having a third fully connected layer where “the number of its nodes is equal to the number of gesture classes,” and concluding with a Softmax classifier to output the final decision about the gesture. Zhang 8.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Justin R. Blaufeld whose telephone number is (571)272-4372. The examiner can normally be reached M-F 9:00am - 4:00pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, James K Trujillo can be reached at (571) 272-3677. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Justin R. Blaufeld
Primary Examiner
Art Unit 2151
/Justin R. Blaufeld/Primary Examiner, Art Unit 2151
Prosecution Insights