DETAILED ACTION
This communication is in response to Application No. 18/043,739 originally filed 03/02/2023. The Request for Continued Examination and Amendment presented on 06/03/2025 which provides amendments to claims 1, 14, 16, 18, 20, 22, 23, 26, Claims 1-10, 12, and 24 are cancelled is hereby acknowledged.
Currently claims 11, 13-23, 25-34 are pending.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/03/2025 has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claim(s) 11, 13-23, 25-34 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Objections
Claims 1 and 23 objected to because of the following informalities: The claims recite “the input object is unknown to the computing unit the computing unit can perform”. It would appear the language requires the use of a “,” for purposes of clarity. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 11, 13-23, 25-28, 31-34 is/are rejected under 35 U.S.C. 103 as being unpatentable over Krenzer et al. WO2020/057993 Published March 26, 2020 hereinafter Krenzer and further in view of Abiko U.S. Patent Application Publication No. 2005/0105782 A1 hereinafter Abiko.
NOTE: As the publication document of Krenzer et al. WO2020/057993, which designates the US, was not originally published in english, for simplicity and ease of citation within this office action, its corresponding US counterpart Krenzer et al. U.S. Patent Application Publication No. 2021/0350105 A1 will be used for citation purposes herein. A copy of the original non-english version of WO2020/057993 has been placed in the file wrapper, and it’s US counterparts U.S. Patent Application Publication No. 2021/0350105 A1 and United States Patent No. 11,580,772 B2 have been made of record and are cited in a former PTO-892 dated 10/25/2023.
Consider Claim 11:
Krenzer discloses a method for controlling a display function of a display apparatus, comprising: (Krenzer, See Abstract.)
capturing, by a camera of a recording apparatus, images of an image sequence of a surface structure of an input object in an input region on a user-facing side of a screen surface of the display apparatus, the input region optically captured from the direction of a side of the screen surface facing away from a user through the screen surface, (Krenzer, [0056], “For the sequence of operations of the method it is first necessary that an application is started, either by the user or by the operating system, or that an application is already running. The term “application” is understood to include programs that have to be started by a user as well as applications started by the operating system. The former kind includes, e.g., e-mail programs, word processors, games, etc.; applications started by the operating system include, e.g., the screen management, which superimposes various symbols known as icons on the screen of the mobile input device; the applications symbolized by them are commonly started by touching them. The prerequisite that an application is started or already running is marked by A) in FIG. 1a. In step B), a finger, or several fingers is/are placed on the screen. In step C), fingerprint reading is activated, and images or image sequences are recorded by means of the fingerprint reader. In the active application, the fingerprint and several fingerprint characteristics are ascertained by the fingerprint reader for at least one finger placed on the screen.”)
and the input object including a surface structure comprising a pattern, the pattern comprising a depth profile; (Krenzer, [0063], “FIG. 3 shows the comparison of a static pattern M with a whole fingerprint F stored in the database. The whole fingerprint—as a rule, only of the first phalanx of one finger—is recorded in the way described above by rolling the flat finger on the fingerprint reader and rolling the finger pulp; therefore, this is a pattern of a relatively large area that in a rough approximation is rectangular, as a rule. A static pattern M recorded by the fingerprint reader represents the part of the whole fingerprint that is currently resting on the display. The ascertainment of the fingerprint and its characteristics or features is carried out as known in prior art and described at the outset. For this purpose, a comparison is made first to find out to which of the whole fingerprints stored it belongs. If this whole fingerprint F exists and has been found, as shown in the example of FIG. 3, the orientation angle α, the rotation angle γ and the setting angle β are calculated on the basis of a pattern comparison, i.e., the section of the whole fingerprint F that corresponds to the static pattern M is identified. As also explained at the outset and known in prior art, the comparison can include the correlation, minutiae, the orientation image, a frequency image, or the shape of the ridges. The position of, e.g., the centroid of the static pattern M in the whole fingerprint F defines the setting angle β and the rotation angle γ; the rotation of the static pattern M relative to the whole fingerprint F defines the orientation angle a. In the example shown, the position of the center of the static pattern M (here, circular) in a coordinate system spanned by the rotation angle γ and the setting angle β corresponds to the respective values that can be assigned to the static pattern M. Here, the error margin for the setting angle β is larger, as its ascertainment varies with the resting force and, therefore, may be less accurate. For ascertaining the setting angle β and the rotation angle γ, it is advantageous to resort also to the shape and size of the static pattern M; wherein, e.g., with a smallish setting angle, the pattern M tends to be largish and of an oblong shape. For individual anatomic reasons, however, the size and shape of the pattern M cannot be used with all fingers or all persons.”)
recognizing, by a computer of the recording apparatus, that the surface structure is in a focus of the display apparatus on the user-facing side of the screen surface; (Krenzer, [0059], “FIG. 1b explains step D) of FIG. 1a in more detail. Step i) shows the initial situation with fingers 3 (here, two fingers 3.1 and 3.2) placed on a screen 1 of a mobile input device 2. The fingerprint reader first records a gray-scale image; a differentiation by colors is unnecessary, because what matters is only the contrast between the skin ridges and skin valleys of the fingerprints. From the gray-scale image shown in step ii), a binary mask image is generated, which is shown in step iii). The mask is used for filtering: Only the intensity values recorded within the white area of the binary mask image shown in step iii) are taken into account; the result is the adjusted graylevel image shown in step iv). Ascertained from this in the further course, as shown in step v) of FIG. 1b and described above, are the static patterns of the fingers, i.e., the sections of the whole fingerprints, and the other fingerprint characteristics such as position, orientation—strictly speaking, orientation angle α, —setting angle β and/or rotation angle γ, and these data are stored in appropriate data structures for further processing.”)
searching, by the computer, an initial image or multiple initial images of the image sequence for the pattern in the surface structure of the input object, wherein the initial image is a first image of the image sequence, in which a surface structure of the input object becomes visible to the recording apparatus; (Krenzer, [0056], “The prerequisite that an application is started or already running is marked by A) in FIG. 1a. In step B), a finger, or several fingers is/are placed on the screen. In step C), fingerprint reading is activated, and images or image sequences are recorded by means of the fingerprint reader. In the active application, the fingerprint and several fingerprint characteristics are ascertained by the fingerprint reader for at least one finger placed on the screen. … The movement pattern results from tracking the position of the finger across a sequence of successive images.”)
when the pattern of the surface structure of the input object is recognized in the respective images of the image sequence, determining, by the computer, an orientation and/or a position of the pattern of the surface structure of the input object in the input region; (Krenzer, [0056], “The fingerprint characteristics include the position of the finger on the screen, the orientation of the at least one finger on the screen, and the rotation angle and/or the setting angle of the at least one finger on the screen. If the at least one finger is moved, a movement pattern is recorded in addition.”)
determining, by the computer, a motion vector of the input object based on a difference in the pattern between the respective orientation and position of the surface structure of the input object between an image and the initial image, wherein when the surface structure of the input object is unknown to the computing unit the computing unit can perform pattern recognition by image processing in the initial image of the surface structure captured by the recording apparatus, the determining of the motion vector comprising: tracking, by the computer, when the input object is tilted and/or inclined and/or rolled away in that a change of the pattern in the images of the image sequence arising in tilting and/or inclination and/or rolling is mapped as tilting and/or rolling and/or inclination of the input object in the motion vector, and (Krenzer, [0003], [0004], [0056], [0061], “Let this approach be explained in more detail with the help of FIGS. 2a, b and FIG. 3. FIG. 2a shows a finger 3 placed on a screen. Here, the screen, in which a fingerprint reader is integrated, is located in the plane spanned by x and y in a Cartesian coordinate system; thus, the direction z corresponds to the normal to the screen. The position of the finger on the screen can be described by its orientation, the setting angle and the rotation angle. The orientation is given relative to a specified axis in the screen plane; in the example shown in FIG. 2a it is given by an orientation angle α in the screen plane, measured relative to an axis y′ parallel to the y-axis. The setting angle β expresses how much the first phalanx of the finger 3 is inclined relative to the screen plane and is measured with reference to the same. The rotation angle γ describes the rotation of the finger 3 about its longitudinal axis. If this rotation is carried out on the screen, this is referred to as “rolling” of the finger 3. Taken together, the three angles unambiguously describe the position or attitude of the respective finger relative to the screen; in principle, any 3-tuple of values can be assigned a function; it should be considered, though, that especially the setting angle β and the rotation angle γ can only be determined with relatively wide error margins, because the resting force can influence the size of the static pattern. If the resting force is recorded in addition, these margins can be narrowed down.”)
combining the respective tilting and/or inclination and/or rolling with a translational movement of the input object in a plane parallel to the user-facing side of the screen surface; and (Krenzer, [0080], [0076], “In CAD programs, components shown on the screen, e.g., can be turned by rotation of the finger, or magnified, demagnified, or zoomed in or out by tilting the finger; rolling the finger can, for example, rotate marked components relative to the screen plane, e.g., about an axis positioned parallel to the screen plane and parallel to the finger resting flat on the screen.”)
controlling, by the computer by the motion vector, the display function of the display apparatus. (Krenzer, [0058], “In step E), finally, the combination of the static pattern of the at least one finger, i.e., of the section of the entire fingerprint, the fingerprint characteristics and, where required, the movement pattern are compared with a database. This is step F) in FIG. 1a. If the combination described is stored in the database, step G) is performed, and a check is made as to whether, for the active application, a specified action is associated with this combination, which will then be carried out; this corresponds to step I). If, whereas the said combination has been stored in the database, it lacks an action associated with it in the active application, a first standard action will be performed, which is step J) in FIG. 1a. If the combination itself is not stored in the database, a second standard action will be performed, being step H) in FIG. 1a. The first and the second standard action may also be identical and, e.g. display an error message indicating that no input is associated with this action.”)
Krenzer however does not appear to expressly suggest wherein when the surface structure of the input object is unknown to the computing unit the computing unit can perform pattern recognition by image processing in the initial image of the surface structure captured by the recording apparatus, such that the pattern of the unknown input object is learned in one or more initial images of the image sequence, such that any objects, which can be optically captured, can be used as input devices, and such that the computer is configured to recognize a pattern from any surface and to determine the motion vector of the input device based on the change of the pattern.
Abiko however teaches that it was a known technique to those having ordinary skill in the art to use a first image as a reference image for than further determining movement and therefore teaches wherein when the surface structure of the input object is unknown to the computing unit the computing unit can perform pattern recognition by image processing in the initial image of the surface structure captured by the recording apparatus, such that the pattern of the unknown input object is learned in one or more initial images of the image sequence, such that any objects, which can be optically captured, can be used as input devices, and such that the computer is configured to recognize a pattern from any surface and to determine the motion vector of the input device based on the change of the pattern. (Abiko, [0150], [0134], “Referring now to the flowchart (step S11 through step S17) of FIG. 20, a description will be made hereinbelow on procedures of movement amount detection and detection error absorption performed by the movement amount detecting means 201. The fingerprint sensor 10 obtains the first image (a partial image) of the fingerprint of the finger 101 (step S11), which image is then set as a reference image for use in detecting an amount of movement of the finger 101 (step S12). After that, the fingerprint sensor 10 obtains the Nth (the initial value of N is 2; N=2, 3, 4, . . . ) fingerprint image (partial image) (step S13), and the Nth fingerprint image and the reference image overlap each other, thereby producing an overlap area to detect a relative positional relationship therebetween, based on which the amount of movement between the reference image and the Nth fingerprint image is then detected (step S14). After that, it is detected whether or not the detected movement amount (the amount in the y-direction in the present example) exceeds a predetermined detection error (step S15).”)
It therefore would have been obvious to those having ordinary skill in the art before the effective filing date of the invention to use the known technique of taking a first image as a reference image for determining movement on the surface of a device as this was known in view of Abiko and would have utilized for the purpose of as would be recognized by a person of ordinary skill in the art such that user convenience can thereby be improved. (Abiko, [0017])
Consider Claim 13:
Krenzer in view of Abiko discloses the method according to claim 11, wherein the determination of the motion vector includes evaluation of a change of a contrast and/or image definition and/or of a scaling of the pattern and/or of a portion of the surface structure of the input object visible for the recording apparatus in the images of the image sequence. (Krenzer, [0025], [0086], [0059], “FIG. 1b explains step D) of FIG. 1a in more detail. Step i) shows the initial situation with fingers 3 (here, two fingers 3.1 and 3.2) placed on a screen 1 of a mobile input device 2. The fingerprint reader first records a gray-scale image; a differentiation by colors is unnecessary, because what matters is only the contrast between the skin ridges and skin valleys of the fingerprints. From the gray-scale image shown in step ii), a binary mask image is generated, which is shown in step iii). The mask is used for filtering: Only the intensity values recorded within the white area of the binary mask image shown in step iii) are taken into account; the result is the adjusted graylevel image shown in step iv). Ascertained from this in the further course, as shown in step v) of FIG. 1b and described above, are the static patterns of the fingers, i.e., the sections of the whole fingerprints, and the other fingerprint characteristics such as position, orientation—strictly speaking, orientation angle α, —setting angle β and/or rotation angle γ, and these data are stored in appropriate data structures for further processing.”)
Consider Claim 14:
Krenzer in view of Abiko discloses the method according to claim 11, wherein the determination of the motion vector includes evaluation of a change of a contrast and/or image definition and/or of a scaling of the pattern and/or of a portion of the surface structure of the input object visible for the recording apparatus in the images of the image sequence. (Krenzer, [0025], [0086], [0059], “FIG. 1b explains step D) of FIG. 1a in more detail. Step i) shows the initial situation with fingers 3 (here, two fingers 3.1 and 3.2) placed on a screen 1 of a mobile input device 2. The fingerprint reader first records a gray-scale image; a differentiation by colors is unnecessary, because what matters is only the contrast between the skin ridges and skin valleys of the fingerprints. From the gray-scale image shown in step ii), a binary mask image is generated, which is shown in step iii). The mask is used for filtering: Only the intensity values recorded within the white area of the binary mask image shown in step iii) are taken into account; the result is the adjusted graylevel image shown in step iv). Ascertained from this in the further course, as shown in step v) of FIG. 1b and described above, are the static patterns of the fingers, i.e., the sections of the whole fingerprints, and the other fingerprint characteristics such as position, orientation—strictly speaking, orientation angle α, —setting angle β and/or rotation angle γ, and these data are stored in appropriate data structures for further processing.”)
Consider Claim 15:
Krenzer in view of Abiko discloses the method according to claim 11, wherein the control of the display function of the display apparatus is effected only after recognizing a stored, authorized pattern of the surface structure of the input object in an image from the image sequence. (Krenzer, [0008], “After ascertainment of the static pattern and the fingerprint characteristics, the combination of the at least one static pattern, the fingerprint characteristics and optionally the at least one movement pattern is compared with a database. Stored in the database are, e.g.,—typical for the user of a mobile phone—several fingerprints, ideally together with various fingerprint characteristics and movement patterns. A differentiation of cases is then made, according to whether or not the said combination is stored in the database. If it is, a check is made to verify that, for the active application, a specified action is associated with this combination, which action is then carried out. If no action is associated with this combination, a first standard action is carried out. If the combination is not stored in the database, a second standard action is carried out. The first and the second standard action may also be identical; for example, a user can get a message that no action is associated with his/her input, and/or a sound signal or a vibration signal can be released.”)
Consider Claim 16:
Krenzer in view of Abiko discloses the method according to claim 11, wherein the control of the display function of the display apparatus is effected only after recognizing a stored, authorized pattern of the surface structure of the input object in an image from the image sequence. (Krenzer, [0008], “After ascertainment of the static pattern and the fingerprint characteristics, the combination of the at least one static pattern, the fingerprint characteristics and optionally the at least one movement pattern is compared with a database. Stored in the database are, e.g.,—typical for the user of a mobile phone—several fingerprints, ideally together with various fingerprint characteristics and movement patterns. A differentiation of cases is then made, according to whether or not the said combination is stored in the database. If it is, a check is made to verify that, for the active application, a specified action is associated with this combination, which action is then carried out. If no action is associated with this combination, a first standard action is carried out. If the combination is not stored in the database, a second standard action is carried out. The first and the second standard action may also be identical; for example, a user can get a message that no action is associated with his/her input, and/or a sound signal or a vibration signal can be released.”)
Consider Claim 17:
Krenzer in view of Abiko discloses the method according to claim 11, wherein at least one display function is each associated with at least one pattern of the input object. (Krenzer, [0008], [0031], “In another preferred embodiment, in which also at least two fingers are simultaneously placed on the screen, for a combination of static patterns, fingerprint characteristics and optionally movement patterns, before a check whether a specific action is associated with this combination, it is checked whether a first of the at least two static patterns corresponds to an activation pattern for activating a multi-fingerprint recording. Only if this is the case will the further check and optionally the release of an action take place. For example, a left-hander may be required to place his/her right thumb at a setting angle of about 45°—with the possibility of stating error tolerances—as an activation pattern. This mode of multi-fingerprint recording after activation prevents inputs being made by mistake on the one hand, and, on the other hand, enables the execution of more or less complex, but frequently used actions with few fingers or finger positions.”)
Consider Claim 18:
Krenzer in view of Abiko discloses the method according to claim 11, wherein at least one display function is each associated with at least one pattern of the input object. (Krenzer, [0008], [0031], “In another preferred embodiment, in which also at least two fingers are simultaneously placed on the screen, for a combination of static patterns, fingerprint characteristics and optionally movement patterns, before a check whether a specific action is associated with this combination, it is checked whether a first of the at least two static patterns corresponds to an activation pattern for activating a multi-fingerprint recording. Only if this is the case will the further check and optionally the release of an action take place. For example, a left-hander may be required to place his/her right thumb at a setting angle of about 45°—with the possibility of stating error tolerances—as an activation pattern. This mode of multi-fingerprint recording after activation prevents inputs being made by mistake on the one hand, and, on the other hand, enables the execution of more or less complex, but frequently used actions with few fingers or finger positions.”)
Consider Claim 19:
Krenzer in view of Abiko discloses the method according to claim 11, wherein the capture of the respective image of the image sequence is effected in combination with a pressure sensor on the screen surface, wherein the pressure sensor measures a pressure of an input by the input object and the capture of the respective image of the image sequence is initiated upon exceeding a threshold value by the pressure or a value of the measured pressure is associated with at least one display function. (Krenzer, [0012], [0042], [0078], [0035], “In yet another embodiment, in which the first of the at least two static patterns corresponds to the activation pattern, a multiple assignment of an input element is implemented by different second static patterns, each of which corresponds to another finger. Here, the term “input element” means an element presented on the screen, e.g., a folder, or an icon of some other application. So far, the common way of implementing this is by different durations, or different pressures, of touching the input element, the results of which are frequently unsatisfactory in practice, as it involves a lot of time, and/or the differentiation of pressures is working less than exactly. The mode of copying text or graphic elements proposed here is faster, more reliable and more convenient than the method used so far, in which, for marking, the touch screen is pressed for about 1 second, after which the element touched by the finger is marked, and subsequently the front and rear boundaries of the marked region can be varied to mark more or less of the content, whereas the distinction of the fingers by the fingerprint allows different functions to be distinguished quickly and precisely, which is quite some gain during inputs by, or interaction with, the user. Herein, multiple assignment of an input element can be implemented not only by using different second static patterns but also by a single second static pattern that corresponds to a fingerprint, with the different assignments then corresponding to different orientations, setting angles and/or rotation angles in the fingerprint characteristics.”)
Consider Claim 20:
Krenzer in view of Abiko discloses the method according to claim 11, wherein the capture of the respective image of the image sequence is effected in combination with a pressure sensor on the screen surface, wherein the pressure sensor measures a pressure of an input by the input object and the capture of the respective image of the image sequence is initiated upon exceeding a threshold value by the pressure or a value of the measured pressure is associated with at least one display function. (Krenzer, [0012], [0042], [0078], [0035], “In yet another embodiment, in which the first of the at least two static patterns corresponds to the activation pattern, a multiple assignment of an input element is implemented by different second static patterns, each of which corresponds to another finger. Here, the term “input element” means an element presented on the screen, e.g., a folder, or an icon of some other application. So far, the common way of implementing this is by different durations, or different pressures, of touching the input element, the results of which are frequently unsatisfactory in practice, as it involves a lot of time, and/or the differentiation of pressures is working less than exactly. The mode of copying text or graphic elements proposed here is faster, more reliable and more convenient than the method used so far, in which, for marking, the touch screen is pressed for about 1 second, after which the element touched by the finger is marked, and subsequently the front and rear boundaries of the marked region can be varied to mark more or less of the content, whereas the distinction of the fingers by the fingerprint allows different functions to be distinguished quickly and precisely, which is quite some gain during inputs by, or interaction with, the user. Herein, multiple assignment of an input element can be implemented not only by using different second static patterns but also by a single second static pattern that corresponds to a fingerprint, with the different assignments then corresponding to different orientations, setting angles and/or rotation angles in the fingerprint characteristics.”)
Consider Claim 21:
Krenzer in view of Abiko discloses the method according to claim 11, wherein the pattern of the input object includes at least one optical marker including at least one pattern for respectively controlling a display function. (Krenzer, [0009], [0058], [0070], [0008], “After ascertainment of the static pattern and the fingerprint characteristics, the combination of the at least one static pattern, the fingerprint characteristics and optionally the at least one movement pattern is compared with a database. Stored in the database are, e.g.,—typical for the user of a mobile phone—several fingerprints, ideally together with various fingerprint characteristics and movement patterns. A differentiation of cases is then made, according to whether or not the said combination is stored in the database. If it is, a check is made to verify that, for the active application, a specified action is associated with this combination, which action is then carried out. If no action is associated with this combination, a first standard action is carried out. If the combination is not stored in the database, a second standard action is carried out. The first and the second standard action may also be identical; for example, a user can get a message that no action is associated with his/her input, and/or a sound signal or a vibration signal can be released.”)
Consider Claim 22:
Krenzer in view of Abiko discloses the method according to claim 11, wherein the pattern of the input object includes at least one optical marker including at least one pattern for respectively controlling a display function. (Krenzer, [0009], [0058], [0070], [0008], “After ascertainment of the static pattern and the fingerprint characteristics, the combination of the at least one static pattern, the fingerprint characteristics and optionally the at least one movement pattern is compared with a database. Stored in the database are, e.g.,—typical for the user of a mobile phone—several fingerprints, ideally together with various fingerprint characteristics and movement patterns. A differentiation of cases is then made, according to whether or not the said combination is stored in the database. If it is, a check is made to verify that, for the active application, a specified action is associated with this combination, which action is then carried out. If no action is associated with this combination, a first standard action is carried out. If the combination is not stored in the database, a second standard action is carried out. The first and the second standard action may also be identical; for example, a user can get a message that no action is associated with his/her input, and/or a sound signal or a vibration signal can be released.”)
Consider Claim 23:
Krenzer discloses a recording apparatus comprising: (Krenzer, See Abstract.)
a camera to capture images of an image sequence of a surface structure of an input object in an input region on a user-facing side of a screen surface of a display apparatus, the input region optically captured from the direction of a side of the screen surface facing away from a user through the screen surface; and (Krenzer, [0066], “In another preferred embodiment, which can also be combined with the layer of angular aperture masks, the illumination for recording the fingerprints is provided by light coupled into a plate-shaped light guide that is arranged above or below the second pixel raster, and the large surfaces of which have light outcoupling elements disposed or shaped on them. Via the light outcoupling elements, the light is coupled out in the direction of the resting surface 4 for the fingers 3 and totally reflected there, unless any skin papillae of a finger are resting on the resting surface 4. Embodiments of this type are shown in FIGS. 5a-5d, where the plate-shaped light guide is symbolized by a luminous layer 8; light is coupled into the light guide laterally through one of its edges. In FIG. 5a, the luminous layer 8 is arranged below the sensor layer 6 with the second pixel raster with sensor elements, but above the display layer 7 with the first pixel raster with image elements. FIG. 5b shows an embodiment in which the luminous layer 8 is arranged between the sensor layer 6 and the display layer 7, with the sensor layer 6 being arranged below the luminous layer 8 as seen in the direction of a viewer. As the light is totally reflected within the luminous layer 8 and can only leave it via the light outcoupling elements (not shown here), the luminous layer 8 may also be arranged above the sensor layer 6 without interfering with the sensitivity of the light-sensitive sensor elements, and no detrimental brightening will take place. FIG. 5c shows another embodiment, in which the two pixel rasters are combined to form a common layer of sensor elements and image elements, and in which the luminous layer 8 is arranged below this common layer. FIG. 5d, finally, shows the common sensor-and-display layer 6, 7 and the luminous layer 8 arranged in an inverted order, with the luminous layer 8 located above the combined sensor-and-display layer 6, 7.”)
the input object including a surface structure comprising a pattern, the pattern comprising a depth profile; (Krenzer, [0063], “FIG. 3 shows the comparison of a static pattern M with a whole fingerprint F stored in the database. The whole fingerprint—as a rule, only of the first phalanx of one finger—is recorded in the way described above by rolling the flat finger on the fingerprint reader and rolling the finger pulp; therefore, this is a pattern of a relatively large area that in a rough approximation is rectangular, as a rule. A static pattern M recorded by the fingerprint reader represents the part of the whole fingerprint that is currently resting on the display. The ascertainment of the fingerprint and its characteristics or features is carried out as known in prior art and described at the outset. For this purpose, a comparison is made first to find out to which of the whole fingerprints stored it belongs. If this whole fingerprint F exists and has been found, as shown in the example of FIG. 3, the orientation angle α, the rotation angle γ and the setting angle β are calculated on the basis of a pattern comparison, i.e., the section of the whole fingerprint F that corresponds to the static pattern M is identified. As also explained at the outset and known in prior art, the comparison can include the correlation, minutiae, the orientation image, a frequency image, or the shape of the ridges. The position of, e.g., the centroid of the static pattern M in the whole fingerprint F defines the setting angle β and the rotation angle γ; the rotation of the static pattern M relative to the whole fingerprint F defines the orientation angle a. In the example shown, the position of the center of the static pattern M (here, circular) in a coordinate system spanned by the rotation angle γ and the setting angle β corresponds to the respective values that can be assigned to the static pattern M. Here, the error margin for the setting angle β is larger, as its ascertainment varies with the resting force and, therefore, may be less accurate. For ascertaining the setting angle β and the rotation angle γ, it is advantageous to resort also to the shape and size of the static pattern M; wherein, e.g., with a smallish setting angle, the pattern M tends to be largish and of an oblong shape. For individual anatomic reasons, however, the size and shape of the pattern M cannot be used with all fingers or all persons.”)
a computer to: recognize that the surface structure is in a focus of the display apparatus on the user-facing side of the screen surface; (Krenzer, [0056], “For the sequence of operations of the method it is first necessary that an application is started, either by the user or by the operating system, or that an application is already running. The term “application” is understood to include programs that have to be started by a user as well as applications started by the operating system. The former kind includes, e.g., e-mail programs, word processors, games, etc.; applications started by the operating system include, e.g., the screen management, which superimposes various symbols known as icons on the screen of the mobile input device; the applications symbolized by them are commonly started by touching them. The prerequisite that an application is started or already running is marked by A) in FIG. 1a. In step B), a finger, or several fingers is/are placed on the screen. In step C), fingerprint reading is activated, and images or image sequences are recorded by means of the fingerprint reader. In the active application, the fingerprint and several fingerprint characteristics are ascertained by the fingerprint reader for at least one finger placed on the screen.”)
search an initial image or multiple initial images of the image sequence for the pattern in the surface structure of the input object, wherein the initial image is a first image of the image sequence, in which a surface structure of the input object becomes visible to the recording apparatus: (Krenzer, [0056], “The prerequisite that an application is started or already running is marked by A) in FIG. 1a. In step B), a finger, or several fingers is/are placed on the screen. In step C), fingerprint reading is activated, and images or image sequences are recorded by means of the fingerprint reader. In the active application, the fingerprint and several fingerprint characteristics are ascertained by the fingerprint reader for at least one finger placed on the screen. … The movement pattern results from tracking the position of the finger across a sequence of successive images.”)
when the pattern of the surface structure of the input object is recognized in the respective images of the image sequence, determine an orientation and/or a position of the pattern of the surface structure of the input object in the input region; (Krenzer, [0056], “. The fingerprint characteristics include the position of the finger on the screen, the orientation of the at least one finger on the screen, and the rotation angle and/or the setting angle of the at least one finger on the screen. If the at least one finger is moved, a movement pattern is recorded in addition.”)
determine a motion vector of the input object based on a difference in the pattern between the respective orientation and position of the surface structure of the input object between an image and the initial image, wherein when the surface structure of the input object is unknown to the computing unit the computing unit can perform pattern recognition by image processing in the initial image of the surface structure captured by the recording apparatus, the determine of the motion vector comprising: track when the input object is tilted and/or inclined and/or rolled away in that a change of the pattern in the images of the image sequence arising in tilting and/or inclination and/or rolling is mapped as tilting and/or rolling and/or inclination of the input object in the motion vector, and (Krenzer, [0003], [0004], [033], [0056], [0061], “Let this approach be explained in more detail with the help of FIGS. 2a, b and FIG. 3. FIG. 2a shows a finger 3 placed on a screen. Here, the screen, in which a fingerprint reader is integrated, is located in the plane spanned by x and y in a Cartesian coordinate system; thus, the direction z corresponds to the normal to the screen. The position of the finger on the screen can be described by its orientation, the setting angle and the rotation angle. The orientation is given relative to a specified axis in the screen plane; in the example shown in FIG. 2a it is given by an orientation angle α in the screen plane, measured relative to an axis y′ parallel to the y-axis. The setting angle β expresses how much the first phalanx of the finger 3 is inclined relative to the screen plane and is measured with reference to the same. The rotation angle γ describes the rotation of the finger 3 about its longitudinal axis. If this rotation is carried out on the screen, this is referred to as “rolling” of the finger 3. Taken together, the three angles unambiguously describe the position or attitude of the respective finger relative to the screen; in principle, any 3-tuple of values can be assigned a function; it should be considered, though, that especially the setting angle β and the rotation angle γ can only be determined with relatively wide error margins, because the resting force can influence the size of the static pattern. If the resting force is recorded in addition, these margins can be narrowed down.”)
combine the respective tilting and/or inclination and/or rolling with a translational movement of the input object in a plane parallel to the user-facing side of the screen surface; and (Krenzer, [0080], [0076], “In CAD programs, components shown on the screen, e.g., can be turned by rotation of the finger, or magnified, demagnified, or zoomed in or out by tilting the finger; rolling the finger can, for example, rotate marked components relative to the screen plane, e.g., about an axis positioned parallel to the screen plane and parallel to the finger resting flat on the screen.”)
control by the motion vector, the display function of the display apparatus. (Krenzer, [0058], “In step E), finally, the combination of the static pattern of the at least one finger, i.e., of the section of the entire fingerprint, the fingerprint characteristics and, where required, the movement pattern are compared with a database. This is step F) in FIG. 1a. If the combination described is stored in the database, step G) is performed, and a check is made as to whether, for the active application, a specified action is associated with this combination, which will then be carried out; this corresponds to step I). If, whereas the said combination has been stored in the database, it lacks an action associated with it in the active application, a first standard action will be performed, which is step J) in FIG. 1a. If the combination itself is not stored in the database, a second standard action will be performed, being step H) in FIG. 1a. The first and the second standard action may also be identical and, e.g. display an error message indicating that no input is associated with this action.”)
Krenzer however does not appear to expressly suggest determine a motion vector of the input object based on a difference in the pattern between the respective orientation and position of the surface structure of the input object between an image and the initial image, wherein when the surface structure of the input object is unknown to the computing unit the computing unit can perform pattern recognition by image processing in the initial image of the surface structure captured by the recording apparatus, such that the pattern of the unknown input object is learned in one or more initial images of the image sequence, such that any objects, which can be optically captured, can be used as input devices, and such that the computer is configured to recognize a pattern from any surface and to determine the motion vector of the input device based on the change of the pattern.
Abiko however teaches that it was a known technique to those having ordinary skill in the art to use a first image as a reference image for than further determining movement and therefore teaches determine a motion vector of the input object based on a difference in the pattern between the respective orientation and position of the surface structure of the input object between an image and the initial image, wherein when the surface structure of the input object is unknown to the computing unit the computing unit can perform pattern recognition by image processing in the initial image of the surface structure captured by the recording apparatus, such that the pattern of the unknown input object is learned in one or more initial images of the image sequence, such that any objects, which can be optically captured, can be used as input devices, and such that the computer is configured to recognize a pattern from any surface and to determine the motion vector of the input device based on the change of the pattern. (Abiko, [0150], [0134], “Referring now to the flowchart (step S11 through step S17) of FIG. 20, a description will be made hereinbelow on procedures of movement amount detection and detection error absorption performed by the movement amount detecting means 201. The fingerprint sensor 10 obtains the first image (a partial image) of the fingerprint of the finger 101 (step S11), which image is then set as a reference image for use in detecting an amount of movement of the finger 101 (step S12). After that, the fingerprint sensor 10 obtains the Nth (the initial value of N is 2; N=2, 3, 4, . . . ) fingerprint image (partial image) (step S13), and the Nth fingerprint image and the reference image overlap each other, thereby producing an overlap area to detect a relative positional relationship therebetween, based on which the amount of movement between the reference image and the Nth fingerprint image is then detected (step S14). After that, it is detected whether or not the detected movement amount (the amount in the y-direction in the present example) exceeds a predetermined detection error (step S15).”)
It therefore would have been obvious to those having ordinary skill in the art before the effective filing date of the invention to use the known technique of taking a first image as a reference image for determining movement on the surface of a device as this was known in view of Abiko and would have utilized for the purpose of as would be recognized by a person of ordinary skill in the art such that user convenience can thereby be improved. (Abiko, [0017])
Consider Claim 25:
Krenzer in view of Abiko discloses a recording apparatus according to claim 23 wherein the determination of the motion vector includes evaluation of a change of a contrast and/or image definition and/or of a scaling of the pattern and/or of a portion of the surface structure of the input object visible for the recording apparatus in the images of the image sequence. (Krenzer, [0025], [0086], [0059], “FIG. 1b explains step D) of FIG. 1a in more detail. Step i) shows the initial situation with fingers 3 (here, two fingers 3.1 and 3.2) placed on a screen 1 of a mobile input device