DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant's amendments filed on January 2, 2025 have been fully considered.
Claim 13 is cancelled.
Claim Objections
Claims 1, 5, 7, 10-11 are objected to because of the following informalities:
Claims 1 and 11: The subject “the captured image” (singular) should read “the captured images” to be consistent to the preceding “captured images” (plural) in line 4.
Claim 5: The subject “the existing position of the head” in line 3 lacks antecedent basis. It is introduced in claim 2, however, claim 5 depends on claim 1 which only introduces “a position of the head” in lines 9-10.
Claim 7: The phrase “that is immediately precedes” should read “that immediately precedes”.
Claim 10: The subject “the one or more object part” should read “the one or more object parts” to be consistent with “one or more object parts” introduced in claim 1.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 6, 8-10, and 12 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Bermano et al (Bermano, A.H., Billeter, M., Iwai, D. and Grundhöfer, A. (2017), Makeup Lamps: Live Augmentation of Human Faces via Projection. Computer Graphics Forum, 36: 311-323. https://doi.org/10.1111/cgf.13128, hereinafter “Bermano”). .
Regarding claim 1,
Bermano teaches:
A simulation system configured to project a projection image onto a head of a living being (Bermano: Page 1, Fig. 1, Abstract, “. . . first system for live dynamic augmentation of human faces. Using projector-based illumination, we alter the appearance of human performers during novel performances. . .”),
the simulation system comprising:
an imager configured to consecutively obtain captured images that contain the head in a time series (Bermano: Page 3, Section 3.1, paragraph 2, “a live performance is captured
by a high-speed camera under IR illumination.”; NOTE: Also see Fig. 9 in page 9, showing consecutively obtained captured images that contain the head. The frames are captured from a live performance of an actor in a continuous and sequential video stream, therefore, the head images obtained are in a time series.);
an information processing unit implemented by a processor (Bermano: Page 1, Abstract, “. . . We have evaluated our system through an optimized CPU and GPU prototype, and demonstrated successful low latency augmentation for different performers and performances with varying facial play and motion speed. . .”)
configured to calculate predetermined information related to the head based on the captured image (Bermano: Page.3, Fig. 2 description, “Figure 2: An overview of our method. A facial rig is deformed and rendered under different illumination conditions. These are stored in an experssion-over-position table. In real-time, a live performer is captured, and the facial expression and configuration are analyzed. The estimated landmark positions are regularized and predicted through a Kalman filter, and are used to deform a 2D mesh. The current time, estimated expression, represented by a reduced set of blendweights, and global position are used to blend the pre-rendered images, and the result is laid over the mesh to create the final augmenting image.”),
and to generate a projection image to be projected (Bermano: Page.3, Fig. 2, NOTE: See the “Projection” image shown in the GPU accelerated block.),
wherein the predetermined information includes a position (Bermano: Page.3, Fig. 2 description, “. . . landmark position . . . global position. . .”),
a shape (Bermano: Page.3, Fig. 2 description, NOTE: The shape of the subject’s face is analyzed using landmark position and used to deform 2D mesh to match the shape of the subject’s head.)
and a projected aspect of the head (Bermano: Page.3, Fig. 2 description, NOTE: The projected aspect of the head is the estimated expression represented by blendweights.),
the projection image to be projected includes a base part and one or more object parts (Bermano: Page 5, Fig. 4, description, “Figure 4: Target appearance representation. An array of nt albedo maps is stored to address temporal changes in skin appearance (left). The neutral pose is rendered for every point is space (bottom right). The spanning of the horizontal axis (n1 samples) is demonstrated. The rest of the nb blendshapes are rendered and stored as offsets from their respective rest pose (top right)”; NOTE: The base part is the neutral pose projection, and the one or more object parts are the array of nt albedo maps blended with a corresponding facial orientation nb.),
and the one or more object parts include a part masking an eye area that is included in the head (Bermano: Page 10, section 5, “. . . Due to the direct projection onto an actor’s face, (eye-)safety is a concern, and warrants an investigation, especially before use in a production setting. For our tests, we used minimal projection brightness, and further applied dark colors to the regions around the eyes. This setting was comfortable for the performer. . .” Page 6, section 3.4, “. . . A set of locations in the image is determined, corresponding to distinct facial features like the eyelids, nose and lips (known as landmarks. . .)”; NOTE: Bermano’s system determines facial landmarks such as eyelids which is an eye area included in the head. The final projection includes applying dark colors to the regions around the eyes. The applied dark colors to the regions around the eyes is the part masking an eye area. Also see Fig. 1, which shows a visible dark color masking the regions around the eyes.);
and a projector (Bermano: Page 7, Section 4, “. . . The main component is a projector-camera system, which is optically aligned using a beam splitter as described in Section 3.2. . .”)
configured to project the projection image generated by the information processing unit in accordance with the position of the head, based on the predetermined information (Bermano: Fig. 1, Page 1 Abstract, “. . . system for live dynamic augmentation of human faces. Using projector-based illumination. . . . detects facial orientation as well as expression. The estimated expression blendshapes are mapped onto a lower dimensional space, and the facial motion and non-rigid deformation are estimated, smoothed and predicted through adaptive Kalman filtering. Finally, the desired appearance is generated interpolating precomputed offset textures according to time, global position, and expression. . . dynamic facial projection mapping. . .”; ),
wherein a time period from a time of a frame when the imager images the head to a time when the projector projects the projection image onto the head, during which the information processing corresponding to this frame is performed, is 50 ms or less (Bermano: Page 1, Abstract, “. . . Therefore, our system aims at reducing latency during every step of the process, from capture, through processing, to projection. . .”; Page 1, Figure 1 description, “. . . The average system latency is 9.8ms, achieved through GPU optimizations, and is further compensated for through prediction. . .”; NOTE: The system latency from capturing the head image, through processing, and projection averages at 9.8 ms, which is less than 50ms.).
Regarding claim 6, depending on 1,
Bermano teaches:
The simulation system according to claim 1,
Bermano further teaches:
wherein the imager is a camera with an operating frequency of 300 fps or more (Bermano: Page 8, Section 4.1, “an Allied Vision Bonito CL-400 B machine vision camera, . . . at a frame rate of 1300 Hz”; NOTE: A frame rate of 1300Hz frequency constitutes 1300 fps, which is more than 300 fps).
Regarding claim 8, depending on 1,
Bermano teaches:
The simulation system according to claim 1,
Bermano further teaches:
wherein the projector is a projection equipment with an operating frequency of 300 Hz or more (Bermano: Page 8, Figure 7 description, “. . . The projector (a) displays 2K images at a frame rate of 480 Hz. . .”).
Regarding claim 9, depending on 1,
Bermano teaches:
The simulation system according to claim 1,
Bermano further teaches:
the simulation system being configured to simulate makeup onto the head (Bermano: Title, “Makeup Lamps: Live Augmentation of Human Faces via Projection”; page 5, section 3.3, paragraph 3, “. . . such as adding makeup, simulating blushing or face painting. . .”; page 2, paragraph 2, “. . . In cosmetics, a user can potentially try different maquillage before physical application. . .”).
Regarding claim 10, depending on 9,
Bermano teaches:
The simulation system according to claim 9,
Bermano further teaches:
further comprising a makeup input device (Bermano: page 8, paragraph 1, . . .The custom-built Intel Xeon workstation. . .”; page 8, section 4.3, “. . . To evaluate our method, we experimented with several sets of textures, configurations and performers. Figure 9a,b,c and d show a few frames of performances where each one was done with a single distinct albedo map. . .”; NOTE: The workstation is the makeup input device because Bermano uses the workstation to select as input the makeup assets, which are the textures (albedo maps constituting to distinct makeup i.e. clown) and project distinct makeups to a face as shown in Figures 9a-9d in page 8),
wherein the information processing unit is configured to output contents of the one or more object part based on an input into the makeup input device (Bermano: page 9, Table 1, “. . . Processing steps timings, over a sequence of 130K input frames. Bold text indicates GPU processing. . .”; NOTE: Also see page 3 Fig. 2, GPU Accelerated block. The GPU is the information processing unit that output the projection contents to a person’s face. In table 1, the task of textured rendering is processed by the GPU.).
Regarding method claim 12,
Method claim 12 is drawn to the method corresponding to the configuration of using same as claimed in system claim 1. Therefore, method claim 1 corresponds to the configuration in the system of claim 1, and is rejected for the same reasons of anticipation as used above.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Bermano.
Regarding claim 7, depending on 1,
Bermano teaches:
The simulation system according claim 1,
Bermano further teaches:
wherein the information processing unit calculates the predetermined information by using a predetermined area centered around the position of the head calculated from a previous frame of the captured image. (Bermano: Page 8, Section 4.2, “For each iteration, the capture thread waits for a signal indicating that a new image is available. reads a small Region Of Interest (ROI) of the image from the frame grabber. This region is updated after each detection according to facial movement. . .”; NOTE: The predetermined area is the ROI of the image. It is centered around the position of head because it is an image of a head. The ROI of the current frame is updated according to a detection of facial movement of a previous frame where the facial movement was detected.)
Although Bermano teaches updating ROI for the current frame from a previous frame, Bermano fails to teach if it is: calculated from a previous frame of the captured image that is immediately precedes a present frame. (NOTE: Bermano updates ROI from a previous frame where facial movement was detected, and possibly but not always necessarily from a frame that immediately precedes a present frame.)
It would have been obvious design choice to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to choose among finite solutions (NOTE: using an immediate frame preceding the present frame / using a previous frame not preceding the present frame which could be 2 or more frames back. Both of which are previous frames to the present frame) to use a predetermined area centered around the position of the head calculated from a previous frame of the captured image calculated from a previous frame of the captured image that is immediately precedes a present frame.
The reason for doing so is for more time efficient ROI processing of the present frame as it only references the immediate preceding frame directly instead of analyzing multiple frames to find a specific frame to be used for ROI update for the present frame.
Claims 2-5, and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Bermano in view of Samain et al. (US 20120029417 A1, hereinafter “Samain”).
Regarding claim 2, depending on 1,
Bermano teaches:
The simulation system according to claim 1,
illuminance of the base part in the projection image to be projected is denoted by I2 [lux] (NOTE: The base part or the neutral pose included in the projection is projected using minimal projection brightness setting described in page 10, section 5, for eye-safety concerns)
Although Bermano teaches limiting the projection brightness to a minimal level as described in page 10, section 5, for eye-safety concerns. Bermano fails to disclose if the threshold limit for the minimal projection brightness is calculated considering the ambient lighting around the subject, and therefore fails to teach: wherein, where illuminance of ambient light at an existing position of the head is denoted by I1 [lux], and illuminance of the base part in the projection image to be projected is denoted by I2 [lux], I2 / I1 is 1.5 or more and 30 or less.
The analogous art Samain teaches adjusting the projector brightness based on the ambient lighting of a subject, including the subject’s head, where the projector brightness denoted by I2, exceeds the ambient light denoted by I1, that is, I2/I1 > 1.0, or I2 > I1(1.0) (Samain: ¶670, “The projection system . . . adjust the intensity of the projection. Where appropriate, adjustment may be automatic by using an optical acquisition device such as a sensor that analyzes ambient light and that adjusts the intensity of the projected light so as to exceed that of ambient light”). Therefore, Samain teaches:
wherein, where illuminance of ambient light at an existing position of the head is denoted by I1 [lux] (Samain: ¶670, “. . . ambient light. . .”),
and illuminance of the base part in the projection image to be projected is denoted by I2 [lux] (Samain: ¶670, “. . . projected light. . .”)
(NOTE: As discussed above, Samain’s system adjusts the projection lighting I2 such that to exceed the ambient light (I1), that is I2/I1 > 1.0, 1.5 is > 0).
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Bermano and Samain to include adaptively adjusting the projection brightness based on the ambient light such that to exceed that of ambient light, and applying Bermano’s eye-safety minimal projection brightness.
The reason for doing so is “to help the user to see the image projected on the face clearly” (Samain: ¶668), and to address eye-safety concerns (Bermano page 10, section 5).
However still, the combination of Bermano’s limiting projection brightness at minimal level for eye-safety and Samain adaptive adjusting of projection brightness does not disclose an exact ambient light to projection light ratio between 1.5 to 30, and therefore fails to teach: I2 / I1 is 1.5 or more and 30 or less. (NOTE: Samain’s I2/I1 > 1.0, while the claim requires I2/I1 = (1.5 – 30))
It would have been obvious design choice to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to try a multiplier for the ambient light such that I2/I1 is greater than 1.0 (Obvious to try multiplier values of 1.0, 1.1, 1.2, . . . 1.5 . . .n) up to a preferred ratio, including 1.5, where it would help the user to see the image projected on the face clearly (NOTE: If the projection brightness has the same brightness as the ambient lighting, the projection image will not be as clear as the projection brightness exceeding the ambient brightness) and limit the projection brightness to a minimum safe level for the eyes as taught by Bermano (NOTE: Obvious to try multiplier values 1.5, 1.6, 1.7 . . .10 . . .20 . . .30. . .40 . . . 50 . . .n. A higher multiplier value makes the projection n times brighter than the ambient light, which when projected to a person’s face will be a safety concern such as glare.) by a factor brighter than the ambient light and select a threshold limit such that the projection brightness is capped at a safe level for the eyes.
The reason for doing so is to limit the projection brightness in consideration for eye safety (Bermano: page 10, section 5).
Regarding claim 3, depending on 2,
The combination of Bermano and Samain teaches:
The simulation system according to claim 2,
Samain further teaches:
wherein the I1 is 500 or less (Samain: ¶686, “The tests were carried out in ambient light in a closed room”; ¶669, “Preferably, the projection takes place while the user is in the dark or in a dimly-lit room, so as to make the projected image stand out better”; NOTE: 500 LUX level can be measured in a bright room. Since, Samain operates in a dark or a dimly-lit room, the LUX level of Samain is inherently below 500 LUX),
Although Samain further teaches the projection takes place in a dark or dimly-lit room which can be as low as 1 lux to 100 lux, and 0 Lux corresponding to complete darkness, Samain is not explicit on exact the Lux values of the dark room whether it is at least 5 LUX, and therefore fails to teach: wherein the I1 is 5 or more.
It would have been obvious design choice to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to try different LUX levels for the ambient lighting minimum starting from 1 LUX, 2, . . . .5 . . .10 . . . 100 and select . Samain operates in a dark or dimly-lit room which has a finite number of LUX level before it is considered to be at bright LUX levels. Selecting a higher LUX level than 0 constituting to a dark room, including 5 LUX is an obvious design choice and obvious to try among finite LUX levels.
The reason for doing so is because selecting a minimum point close to 0 LUX will make the room very dark that it will be difficult for people in the room to navigate the room physically, having a slightly higher LUX level still within the dimly-lit room range can improve safety within the room.
Regarding claim 4, depending on 2,
The combination of Bermano and Samain teaches:
The simulation system according to claim 2,
Bermano further teaches:
wherein, where illuminance of an area corresponding to a pupil part in the projection image to be projected by the projector is denoted by I3 [lux], I3/I1 is 10 or less (Bermano: page 10, section 5, “. . . minimal projection brightness, and further applied dark colors to the regions around the eyes. . .”; NOTE: Projecting dark colors to the regions around the eyes, including a pupil part, constitutes to a LUX level of close to 0 LUX. Also see figure 9, where the projection is illustrated showing the dark regions around the eyes which are visibly all constitute to low LUX level, which is less brighter than the rest of the projection and ambient lighting. Therefore, the illuminance of the eye region I3 in relation to the ambient lighting I1 is inherently less than 1 (which is less than 10 as claimed). That is I3/I1 < 1.0).
Regarding claim 5, depending on 1,
Bermano teaches:
The simulation system according to claim 1,
However, Bermano fails to teach the system: further comprising an illuminance adjustment unit configured to adjust illuminance of ambient light at the existing position of the head by a member that is arranged around the head.
The analogous art Samain teaches:
further comprising an illuminance adjustment unit configured to adjust illuminance of ambient light at the existing position of the head by a member that is arranged around the head (Samain: ¶171-173, “. . . The optional optical acquisition device, the irradiator, the optional computer, and the optional view screen may be produced in the form . . . integrated in the same casing. . . The face may then be moved close to be placed on the casing, e.g. by leaning. . .”; NOTE: The casing acts as the illuminance adjustment unit arranged round the head when the face is placed or moved close to the casing by leaning. It adjust the illuminance of ambient light by physically blocking the room ambient light towards the face. By blocking the room ambient light, it inherently adjusts the ambient light at the existing position of the head).
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to combine Bermano and Samain to include: an illuminance adjustment unit configured to adjust illuminance of ambient light at the existing position of the head by a member that is arranged around the head.
The reason for doing so is because the casing where the face is placed provides stability “to reduce the risk of blurring the resulting light-sensitive makeup look” (Samain: ¶45) and “to make the projected image stand out better” (Samain: ¶669).
Regarding claim 11,
Claim 11 is drawn to a system which have similar in scope to the simulation system of Claim 2, without the eye masking feature and the 50ms latency feature and is rejected for the same reasons of obviousness as used above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PATRICK GALERA whose telephone number is (571)272-5070. The examiner can normally be reached Mon-Fri 0800-1700 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PATRICK P GALERA/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights