Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Examiner Note
Claims were filed 5/20/24 along with amended claims filed 5/20/24. The unamended claims (along with the original specification and abstract) were refiled 10/10/24. The claims refiled 10/10/24 do not include any markup and are non-compliant and appears to be unintended. The 5/20/24 claims with markup are therefore being used.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Objections
Claim 6 is objected to because of the following informalities: the claims recites “wherein the second storage device includes a first storage device and a second storage device”. Since a storage device is already recited in claim 1, the “second” appears to be a typographical error and the claim should be “wherein the .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-7 is/are rejected under 35 U.S.C. 102(a)(1) as being clearly anticipated by Cosatto U.S. Patent/PG Publication 6504546.
Regarding claim 1 (independent):
A display control device comprising a storage device and an execution device, wherein (Cosatto C3 L20-25 computer graphic application) (Cosatto C4 L15-20 easy access in a database.)(Cosatto C11 L50-55 storage)
the storage device stores 2D texture data that is a plurality of texture data related to an agent, (Cosatto C5 L30-35 The base face covers the area of the whole face and serves as a substrate onto which the facial parts (the remaining three-dimensional planes, as discussed below) will be integrated)(Cosatto C5 L55-65 For the base face, bitmap samples are recorded with the head in different orientations. One exemplary range of head rotations that may be used in practicing the present invention is .+-.15.degree., although it is to be understood that any suitable range of rotation may be used, with additional bitmap samples recorded if a larger range of head rotations is utilized.) where the base face is a plane and therefore a 2D texture (see fig. 1).
and 3D texture data that is data different from the 2D texture data and is data including texture data of an eye portion and mouth portion of the agent, (Cosatto C10 L25-35 For facial parts, single images are primarily used. Since the recognition module provides extensive information about the shape of facial features, it is possible to reliably parameterize them. For cases where the appearance of the facial part cannot be properly described by the chosen parameters, e.g. a smiling mouth, short sequences are stored, and labeled with their appearance. For the base face, short sequences of typical head movements are also stored.)(Cosatto C11 L45-55 In the exemplary embodiment described above, the mouth shape is characterized by three parameters (width, highest point of upper lip and lowest point of lower lip), dividing them into four, four and three intervals, respectively, resulting in a database of 48 mouth samples. About 40 additional samples are necessary to store the remaining facial parts (jaw, eyes, forehead & eyebrows).) where the mouth and eyes/eyebrows are both separate planes (creating a 3D shape) and used to illustrate the 3D shape of the model (Cosatto C5 L45-50) and are therefore 3D textures.
the agent is a person who interacts with a user, (Cosatto C3 L50-55 The result is a talking head that resembles very closely the person who was originally recorded.)
the 2D texture data includes a plurality of pieces of data indicating mutually different postures of the agent, (Cosatto C5 L55-65 For the base face, bitmap samples are recorded with the head in different orientations. One exemplary range of head rotations that may be used in practicing the present invention is .+-.15.degree., although it is to be understood that any suitable range of rotation may be used, with additional bitmap samples recorded if a larger range of head rotations is utilized.)
the 3D texture data includes a plurality of pieces of data of an eye portion and mouth portion indicating each of mutually different facial expressions of the agent, (Cosatto C6 L5-10 To generate a face with a certain mouth shape and emotional expression, the proper bitmaps are chosen for each of the facial parts.) (Cosatto C11 L45-55 In the exemplary embodiment described above, the mouth shape is characterized by three parameters (width, highest point of upper lip and lowest point of lower lip), dividing them into four, four and three intervals, respectively, resulting in a database of 48 mouth samples. About 40 additional samples are necessary to store the remaining facial parts (jaw, eyes, forehead & eyebrows).)
the execution device executes 2D mapping processing, 3D mapping processing, and display processing, the 2D mapping processing is processing of mapping, to a 2D model, a piece of data selected from among a plurality of posture data included in the 2D texture data, the 3D mapping processing is processing of mapping, to a 3D model (Cosatto C3 L25-35 method for modeling three-dimensional objects and, more particularly, to modeling three-dimensional objects using a data-driven approach with separate three-dimensional image planes so as to synthesize three-dimensional, photo-realistic animations.), a piece of data of a facial expression selected from among a plurality of facial expressions included in the 3D texture data, and (Cosatto C6 L5-20 To generate a face with a certain mouth shape and emotional expression, the proper bitmaps are chosen for each of the facial parts. The head orientation is known from the base face, so the bitmaps can be projected onto the base face using simple warping, as shown in FIG. 5. This operation is similar to traditional texture mapping. The difference with traditional 3D modeling techniques is that for plastic deformations, different bitmaps are selected, rather than trying to squeeze one single bitmap into any new shape. Only rigid movements such as rotation and translation of the whole head plus the rotation of the jaw are modeled. The bitmaps of the facial parts are integrated into the base face with proper "feathering" (i.e., alpha-blending at the edges), so that they blend more smoothly into the base face without introducing artifacts (see FIG. 6).)
the display processing is processing of displaying, on a display unit, image data obtained by combining image data subjected to the 2D mapping processing with image data subjected to the 3D mapping processing (Cosatto C14 L60-67 A frame of the final animation can be generated when bitmaps of all the face parts have been retrieved from the database. The bitmap of the base face is first copied into the frame buffer, then the bitmaps of face parts are projected onto the base face using the 3D model and the pose.).
Regarding claim 2:
The display control device according to claim 1, has all of its limitations taught by Cosatto. Cosatto further teaches wherein the 3D texture data includes a plurality of pieces of data of an eyebrow portion, the plurality of pieces of data indicating each of mutually different facial expressions of the agent (Cosatto C10 L25-35 About 40 additional samples are necessary to store the remaining facial parts (jaw, eyes, forehead & eyebrows).)).
Regarding claim 3:
The display control device according to claim 1, has all of its limitations taught by Cosatto. Cosatto further teaches wherein the storage device stores 3D model data to which the 3D texture data is mapped, data of a mouth portion among the 3D texture data includes data of a jaw portion, (Cosatto C11 L45-55 In the exemplary embodiment described above, the mouth shape is characterized by three parameters (width, highest point of upper lip and lowest point of lower lip), dividing them into four, four and three intervals, respectively, resulting in a database of 48 mouth samples. About 40 additional samples are necessary to store the remaining facial parts (jaw, eyes, forehead & eyebrows). Each sample is about 5 kB (compressed using JPEG), resulting in about 1/2 MB of storage.)
data of the jaw portion includes data of a state where a mouth is closed and data of a state where a mouth is open, (Cosatto C6 L50-55 A person is recorded while freely speaking a few short sentences to obtain all of the different mouth shapes.)
and the 3D model to which data of the jaw portion in a state where the mouth is closed is mapped and the 3D model to which data of the jaw portion in a state where the mouth is open is mapped are the same model (Cosatto Fig. 3-5) where the mouth/jaw planes form a model which are the same for the open and closed mouth.
Regarding claim 4:
The display control device according to any one of claims 1, wherein the storage device stores specification data that is data in which a position and rotation angle of the 3D model are specified for each of mutually different postures of the agent, the postures being indicated by the 2D texture data, and the 3D mapping processing is processing of mapping the 3D texture data to the 3D model on the basis of the specification data (Cosatto C10 L35-50 From the recognition module, the position and shape of the facial parts, as well as the pose of the whole head, is known (as shown in FIG. 18). To extract facial parts from the image, the planes of the 3D model are first projected into the image plane (illustrated in FIG. 19). The projected planes then mark the extent of each facial feature (see FIG. 20). These areas are "un-warped" into the normalized bitmaps of FIG. 21. Any information about the shape produced by the recognition module is also mapped into the normalized view and stored along with the bitmap in a data structure.)(Cosatto C15 L20-30 It is to be understood that the simple 3D model currently used covers a limited range of views. This is because it approximates facial parts with only a few planes, resulting in visible artifacts when the head is rotated beyond about +15.degree. of the original sample's angle. To circumvent this limitation, the model can be augmented with new sets of samples that are extracted under different views. In this way, a wider range of possible views can be covered by switching between sets of samples depending on the pose of the base head.)(Cosatto C3 L45-50 These trajectories provide the parameters for selecting the proper bitmaps from the database.)(Cosatto C4 L15-20 The needed bitmaps are then extracted from the video frames and normalized and parameterized for easy access in a database.)
Regarding claim 5:
The display control device according to claim 1, has all of its limitations taught by Cosatto. Cosatto further teaches wherein the 3D texture data stored in the storage device is pre-rendered data (Cosatto C10 L35-60 Before the image samples are entered into the database, they are corrected in shape and scale to compensate for the different head orientations when they were recorded. From the recognition module, the position and shape of the facial parts, as well as the pose of the whole head, is known (as shown in FIG. 18). To extract facial parts from the image, the planes of the 3D model are first projected into the image plane (illustrated in FIG. 19). The projected planes then mark the extent of each facial feature (see FIG. 20). These areas are "un-warped" into the normalized bitmaps of FIG. 21. Any information about the shape produced by the recognition module is also mapped into the normalized view and stored along with the bitmap in a data structure. For example, the recognition module produces the outline of lips encoded as a sequence of points. All these points are then mapped into the normalized plane before entering them into the database. Once all of the samples of a face part are extracted from the video sequences and normalized, they need to be labeled and sorted in a way that allows for efficient retrieval.).
Regarding claim 6:
The display control device according to any one of claims 1, wherein the second storage device includes a first storage device and a second storage device,
the first storage device stores the 3D texture data at all times, (Cosatto C4 L15-20 The needed bitmaps are then extracted from the video frames and normalized and parameterized for easy access in a database.) where the database is necessarily part of a device since it is stored data.
the execution device executes write processing, the write processing is processing of writing, to the second storage device, the 3D texture data stored in the first storage device, and the 3D mapping processing is processing of mapping the 3D texture data to the 3D model by using the 3D texture data written to the second storage device (Cosatto C13 L60-67 A frame of the final animation can be generated when bitmaps of all the face parts have been retrieved from the database. The bitmap of the base face is first copied into the frame buffer, then the bitmaps of face parts are projected onto the base face using the 3D model and the pose.).
Regarding claim 7:
The display control device according to claim 1, has all of its limitations taught by Cosatto. Cosatto further teaches wherein the execution device executes 2D selection processing of selecting, according to an interaction with the user, (Cosatto C1 L15-40 With respect to learning tasks, several researchers have reported that animated characters can increase the attention span of the user, and hence improve learning results. When used as avatars, lively talking heads can make an encounter in a virtual world more engaging. […] An animated face of a competent teacher, on the other hand, can create an atmosphere conducive to learning and therefore increase the impact of such educational software.) data to be utilized for the 2D mapping processing, from among a plurality of pieces of data indicating mutually different postures of the agent, (Cosatto C11 L15-20 Searching through all samples, each grid point can be populated with the k closest candidate bitmaps.)(Cosatto C12 L35-40 As mentioned above, a text-to-speech (TTS) synthesizer provides the audio track as well as a phoneme string for animation, where the synthesizer calculates motion trajectories for all of the facial parts, as well as the base face. These trajectories provide the parameters for selecting the proper bitmaps from the database.)
and 3D selection processing of selecting, according to an interaction with the user, (Cosatto C1 L15-40 With respect to learning tasks, several researchers have reported that animated characters can increase the attention span of the user, and hence improve learning results. When used as avatars, lively talking heads can make an encounter in a virtual world more engaging. […] An animated face of a competent teacher, on the other hand, can create an atmosphere conducive to learning and therefore increase the impact of such educational software.) data to be utilized for the 3D mapping processing, from among a plurality of pieces of data indicating each of mutually different facial expressions of the agent (Cosatto C11 L15-20 Searching through all samples, each grid point can be populated with the k closest candidate bitmaps.)(Cosatto C12 L35-40 As mentioned above, a text-to-speech (TTS) synthesizer provides the audio track as well as a phoneme string for animation, where the synthesizer calculates motion trajectories for all of the facial parts, as well as the base face. These trajectories provide the parameters for selecting the proper bitmaps from the database. Smoothing and blending may be applied to these "strings" of bitmaps to eliminate hard transitions and create a seamless animation for each facial part.)
Conclusion
For the prior art referenced and the prior art considered pertinent to Applicant’s disclosure but not relied upon, see PTO-892 “Notice of References Cited”.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASON PRINGLE-PARKER whose telephone number is (571) 272-5690 and e-mail is jason.pringle-parker@uspto.gov. The examiner can normally be reached on 8:30am-5:00pm est Monday-Friday. If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, King Poon can be reached on (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, seehttp://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASON A PRINGLE-PARKER/
Primary Examiner, Art Unit 2617
Prosecution Insights