DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
1 This action is in response to the amendment filed on 03/13/2026. Claims 1-14, 16-17, 19-22, and 24-25 have been amended, and claims 18 and 23 have been cancelled to overcome an objection. Claims 1-14, 16-17, 19-22, and 24-25 remain rejected.
Response to Arguments
2 Applicant’s arguments with respect to claims 1, 8, 16, and 21 filed on 03/13/2026, with respect to the rejection under 35 U.S.C. § 102 regarding that the prior art does not teach the following but not limited to “…acquiring a target image representing a previously captured image of the object;… determining a transform matrix based on a color space in each of the live input image and the target image corresponding to the shared reference region; transforming at least a portion of the acquired live input image by applying the transform matrix to relight the live input image;”. This argument has been considered, but are moot due to new grounds of rejection under 35 U.S.C. § 103, alongside more detail surrounding existing rejections. Specifically, the “targets” mentioned in the prior may be true that it represents the specific places in a room, however they are still visual references, and the certain systems that can be taught will still view those targets for that image as if it was reading the image in general. Further prior art that supports the amended parts of the “target image” is discussed in the rejection below.
3 Regarding claims 2-14, 17, 19-20, 22, and 24-25, they directly/indirectly depend on independent claims 1, 8, 16, and 21 respectively. Applicant does not argue anything other than independent claims 1, 8, 16, and 21. The limitations in those claims, in conjunction with combination, was previously established as explained.
4 Claims 18 and 23 have been cancelled as mentioned previously, therefore they will not be reviewed further.
Claim Rejections - 35 USC § 103
5 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
6 The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
7 Claim(s) 1-2, 5, 8-9, 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Williamson et al. (US 20040104935 A1) in view of Yildiz et al. (US 10540812 B1).
8 Regarding claim 1, Williamson teaches a method for relighting captured images in a virtual reality environment ([0158] reciting “A more attractive approach to fast 3D model construction is shape-from-silhouette. A number of cameras are placed around the subject. Each pixel in each camera is classified as either belonging to the subject (foreground) or the background. The resulting foreground mask is called a "silhouette". Each pixel in each camera collects light over a (very narrow) rectangular-based pyramid in 3D space, where the vertex of the pyramid is at the focal point of the camera and the pyramid extends infinitely away from this.”; [0180] reciting “Third, unless the bi-directional reflectance distribution function is uniform, the actual reflected light will vary at different camera vantage points.”; [0327] reciting “The detection algorithm thresholds the intensity of the full-color video image before doing the preliminary stages of target detection. This is done using a fixed brightness threshold…”), the method comprising: acquiring a live input image from a camera including an object ([0058] reciting “Virtual Viewpoint Remote Collaboration consists of a series of simulation booths equipped with multiple cameras observing the participants' actions. The video from these cameras is captured and processed in real-time to produce information about the three-dimensional structure of each participant. From this 3D information, Virtual Viewpoint technology is able to synthesize an infinite number of views from any viewpoint in the space, in real-time and on inexpensive mass-market PC hardware.”; [0065] reciting “A method for determining the 3D structure of the human form or object in real-time. Any of a number of methods can be used. In order to control the cost of the systems, several methods have been developed which make use of the images from the cameras in order to determine 3D structure.”; [0257] reciting “Once the approximate 3D shape of objects are known, a number of methods can be used to create novel 3D views of the object using the input video. These methods fall into a broad class known as Image-Based Rendering (IBR).”);
acquiring a target image ([Abstract] reciting “A plurality of target markers are distributed within a virtual environment room where each target is distinct from all other targets in the virtual environment room and distinct from rotated versions of itself. An automatic calibration program selects pair of targets from an image from the video camera.”; [0297] reciting “The image digitizer 1702 signal is used for detection of targets within the camera view 1703. The targets within the camera view are detected 1704.”; [0305] reciting “The pattern inside the target can be thought of as an image at some particular resolution.”)
9 Although Williamson could teach determining at least one shared reference region based on a common area of the object in each of the live input image and the target image ([0172] reciting “It is assumed that the subject is completely visible in every image. To constrain the search for each virtual pixel, the corresponding ray is intersected with the boundaries of each image. The ray is projected into each real image to form the corresponding epipolar line. The points where these epipolar lines meet the image boundaries are found and these boundary points are projected back onto the ray. The intersections of these regions on the ray define a reduced search space.”); determining a transform matrix based on a color space in each of the live input image and the target image corresponding to the shared reference region ([0165] reciting “Given any standard 4.times.4 projection matrix representing the desired virtual camera, the center of each pixel of the virtual image is associated with a ray in space that starts at the camera center and extends outward. Any given distance along this ray corresponds to a point in 3D space. In order to determine what color to assign to a particular virtual pixel, the first (closest) potentially occupied point along this ray must be known. This 3D point can be projected back into each of the real cameras to obtain samples of the color at that location.”; [0197] reciting “For each frame, the augmented reality system identifies the transformation matrix relating marker and camera positions. This is passed to the virtual viewpoint server, together with the estimated camera calibration matrix. The server responds by returning a 374.times.288 pixel, 24 bit color image, and a range estimate associated with each pixel.”); transforming at least a portion of the acquired live input image by applying the transform matrix to relight the live input image ([0097] reciting “This matrix takes the 3D homogeneous coordinate in space and converts it into an image-centered coordinate.”); outputting the transformed portion of the acquired live input image as a relit image ([0321] reciting “The target pair processing is repeated until all target pairs in the image are processed 2112. If the captured image has completed the processing stage 2113, then the optimal set of transforms is calculated 2114 and the set of targets and transforms are output 2115.”); and displaying, in a virtual environment, the relit image of the acquired live input image ([0197] reciting “For each frame, the augmented reality system identifies the transformation matrix relating marker and camera positions. This is passed to the virtual viewpoint server, together with the estimated camera calibration matrix. The server responds by returning a 374.times.288 pixel, 24 bit color image, and a range estimate associated with each pixel. This simulated view of the remote collaborator is then superimposed on the original image and displayed to the user.”), prior art from Yildiz can teach this limitation further. Williamson does not explicitly teach acquiring a target image representing a previously captured image of the object…
10 Yildiz teaches acquiring a target image representing a previously captured image of the object ([Page 11; Column 2, Lines 51-55] reciting “In some cases, the method may include detecting movement of the real-world light source; simulating a corresponding movement of the virtual light source; and updating an illumination of the overlay image according to the movement.”; [Page 12; Column 4, Lines 48-51] reciting “In some cases, an IR or NIR SLAM camera may be further configured to capture thermographic images of target objects, and to provide accurate non-contact temperature measurements of those objects.”; [Page 14; Column 8, Lines 45-50] reciting “The map may be maintained using predictions (e.g., when HMD 102 moves) and/or corrections (e.g., camera 108 observes landmarks in the environment that have been previously mapped). In other cases, a map of environment 100A may be obtained, at least in part, from cloud 104.”);
determining at least one shared reference region based on a common area of the object in each of the live input image and the target image ([Page 15; Column 10, Lines 33-36] reciting “In some cases, gaze tracking module 405 may be configured to identify a direction, extent, and/or speed of movement of the user's eyes in real-time, during execution of an xR application (e.g., a gaze vector).”; [Page 16; Column 11, Lines 22-25] reciting “Scanpath: a series of short fixations and saccades alternating before the eyes reach a target location on the screen (e.g., scanpath direction, duration, length and area covered);”; [Page 17; Column 14, Lines 40-45] reciting “At block 601, method 600 may begin calibration using images received from camera 108 mounted on HMD 102, for example. At block 602, method 600 may prompt user 101 wearing HMD 102 to rotate in position around the entire room, to cover an entire spherical area.”);
determining a transform matrix based on a color space in each of the live input image and the target image corresponding to the shared reference region ([Page 12; Column 3, Lines 7-11] reciting “To illuminate or shade the image rendered using the property, the IHS may be further configured to transform a real-world location of the real-world light source to another location in a coordinate system used by the AR application.”; [Page 16; Column 12, Lines 29-33] reciting “The inverse of the view matrix is referred to as the camera transform matrix, which describes how camera 108 itself moves around a scene or frame. That is, the camera transform matrix provides the position and rotation of camera 108.”; [Page 17; Column 14, Lines 51-58] reciting “For example, after reading an image frame, block 604 may transform RGB channels into luminance for each pixel. Next, block 604 searches the image for local maximums with a Monte-Carlo method. Block 604 selects the point with largest luminance from the set, and the process may be repeated several times. Then, block 604 finds a group of points that belong to a single area source of light.”);
transforming at least a portion of the acquired live input image by applying the transform matrix to relight the live input image ([Page 11; Column 2, Lines 18-27] reciting “The program instructions may further cause the IHS to transform a real-world location of the real-world light source to a virtual location in a coordinate system used by the xR application. Additionally, or alternatively, the program instructions may further cause the IHS to classify the real-world light source as: point, spot, directional, or area. To indicate the property, the program instructions may further cause the IHS to build a look-up table (LUT) of virtual light sources by type, position, color, intensity, and size.”; [Page 11; Column 2, Lines 51-59] reciting “In some cases, the method may include detecting movement of the real-world light source; simulating a corresponding movement of the virtual light source; and updating an illumination of the overlay image according to the movement. The method may also include detecting a change in intensity of the real-world light source; simulating a corresponding change of intensity of the virtual light source; and updating an illumination of the overlay image according to the change in intensity.”; [Page 16; Column 11; Lines 58-63] reciting “STAR module 408 may operate with xR application 401 to track the position or movement of light sources in real-time, or near real-time using SLAM landmarks. Moreover, STAR module 408 may be configured to provide a software service described in FIG. 5.”);
outputting the transformed portion of the acquired live input image as a relit image ([Page 15; Column 9, Lines 44-48] reciting “The observed data and annotations may then be used to generate one or more machine-learned algorithms that map inputs (e.g., observation data from a depth camera) to desired outputs (e.g., body-part indices for relevant pixels).”; ([Page 11; Column 2, Lines 18-27] reciting “The program instructions may further cause the IHS to transform a real-world location of the real-world light source to a virtual location in a coordinate system used by the xR application. Additionally, or alternatively, the program instructions may further cause the IHS to classify the real-world light source as: point, spot, directional, or area. To indicate the property, the program instructions may further cause the IHS to build a look-up table (LUT) of virtual light sources by type, position, color, intensity, and size.”; [Page 11; Column 2, Lines 51-59] reciting “In some cases, the method may include detecting movement of the real-world light source; simulating a corresponding movement of the virtual light source; and updating an illumination of the overlay image according to the movement. The method may also include detecting a change in intensity of the real-world light source; simulating a corresponding change of intensity of the virtual light source; and updating an illumination of the overlay image according to the change in intensity.”);
and displaying, in a virtual environment, the relit image of the acquired live input image ([Abstract] reciting “cause the IHS to: detect a real-world light source using a sensor mounted on a Head-Mounted Device (HMD); identify a property of the real-world light source; and indicate the property to a rendering engine during execution of an xR application, where the rendering engine is configured to render an image for display by the HMD based on the property.”; [Page 13; Column 5, Lines 19-26] reciting “…create artificial light sources in a virtual world's coordinates that correspond to the real-world light sources, and periodically update the light sources as part of the rendering of virtual objects to be displayed by HMD 102. These systems and methods may periodically update light sources during the course of execution of the xR application, in response to movement, brightness, or color changes, etc.”).
11 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Williamson) to incorporate the teachings of Yildiz to provide a clearer method regarding determining when an image is previously captured, determining a shared reference region, finding a transformation matrix, as well as outputting and displaying the live input images, utilizing the relit methods taught by Williamson. Doing so would include certain types of tracking including inertial and optical tracking as stated Yildiz ([Page 12; Column 4, Lines 26-37] recited).
12 Regarding claim 2, Williamson in view of Yildiz teaches the method according to claim 1 (see claim 1 rejection above), and although Williamson could teach wherein the determination of at least one shared reference region for the acquired live input image and at least one shared reference region for the acquired target image is based on respective extracted feature points on the acquired live input image and the acquired target image ( [0065] reciting “A method for determining the 3D structure of the human form or object in real-time. Any of a number of methods can be used. In order to control the cost of the systems, several methods have been developed which make use of the images from the cameras in order to determine 3D structure.”; [0257] reciting “Once the approximate 3D shape of objects are known, a number of methods can be used to create novel 3D views of the object using the input video. These methods fall into a broad class known as Image-Based Rendering (IBR).”; [0077] reciting “The LEDs are activated one at a time so that any video image of the plane will have a single bright spot in the image. By capturing 64 images from each camera, each LED is imaged once by each camera. By sequencing the LEDs in a known order, software can determine the precise 3D position of the LED. Finally, by elevating the plane to different heights, a set of points in 3 dimensions can be acquired.”), prior art from Yildiz from claim 1 can teach this limitation further.
13 Yildiz teaches wherein the determination of at least one shared reference region for the acquired live input image and at least one shared reference region for the acquired target image is based on respective extracted feature points on the acquired live input image and the acquired target image ([Page 15; Column 10, Lines 33-36] reciting “In some cases, gaze tracking module 405 may be configured to identify a direction, extent, and/or speed of movement of the user's eyes in real-time, during execution of an xR application (e.g., a gaze vector).”; [Page 16; Column 11, Lines 22-25] reciting “Scanpath: a series of short fixations and saccades alternating before the eyes reach a target location on the screen (e.g., scanpath direction, duration, length and area covered);”; [Page 17; Column 14, Lines 40-45] reciting “At block 601, method 600 may begin calibration using images received from camera 108 mounted on HMD 102, for example. At block 602, method 600 may prompt user 101 wearing HMD 102 to rotate in position around the entire room, to cover an entire spherical area.”; [Page 17; Column 13, Lines 62-67] reciting “The ML engine learns what features correspond to what labels, which enables the classifier to predict the class of future light sources. Features that are extracted from the images may include: the outline of the light source, shape, light intensity, perceived direction/angle, light color, distance from the light source, etc.”).
14 As explained in the rejection of claim 1, the obviousness for combining of a shared reference region of Yildiz into Williamson is provided above, as well as Yildiz’s teachings of the extracted features from these regions, utilizing the input images taught by Williamson. Doing so would allow the distinguishing between point and directional lights as stated by Yildiz ([Page 17; Column 14, Lines 4-7] recited).
15 Regarding claim 5, Williamson in view of Yildiz teaches the method according to claim 1 (see claim 1 rejection above), wherein acquiring the live input image and acquiring the target image is performed via a manual operation ([0132] reciting “They can talk to one another, see each other's actual clothing and actions, all in real-time. They can walk around one another, move about in the virtual room and view each other from any angle. Participants enter and experience simulations from any viewpoint and are immersed in the simulation.”; [0309] reciting “The test application automatically outputs the detection results including the confidence of each detection, target size, etc. The test application allows manual verification of selected targets.”).
16 Claim 8 has similar limitations as of claim 1, therefore it is rejected under the same rationale as claim 1.
17 Claim 9 has similar limitations as of claim 2, therefore it is rejected under the same rationale as claim 2.
18 Claim 12 has similar limitations as of claim 5, therefore it is rejected under the same rationale as claim 5.
19 Claim(s) 3-4 and 10-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Williamson et al. (US 20040104935 A1) in view of Yildiz et al. (US 10540812 B1) as of claim 1, further in view of Mori et al. (US 20100157341 A1).
20 Regarding claim 3, Williamson in view of Yildiz teaches the method according to claim 1 (see claim 1 rejection above), and Yildiz from claim 1 teaches further comprising on at least one shared reference region for the acquired live input image and the at least one shared reference region for the acquired target image ([Page 15; Column 10, Lines 33-36] reciting “In some cases, gaze tracking module 405 may be configured to identify a direction, extent, and/or speed of movement of the user's eyes in real-time, during execution of an xR application (e.g., a gaze vector).”; [Page 16; Column 11, Lines 22-25] reciting “Scanpath: a series of short fixations and saccades alternating before the eyes reach a target location on the screen (e.g., scanpath direction, duration, length and area covered);”; [Page 17; Column 14, Lines 40-45] reciting “At block 601, method 600 may begin calibration using images received from camera 108 mounted on HMD 102, for example. At block 602, method 600 may prompt user 101 wearing HMD 102 to rotate in position around the entire room, to cover an entire spherical area.”; [Page 17; Column 14, Lines 9-16] reciting “The observed color of objects in the image may be dependent on the intrinsic color of the object, how reflective the object is, illuminance from light sources and relative position of the object to light sources. A similar approach may be used to determine light type, where directionality of light sources are determined by observing the other objects around. ”). As explained in the rejection of claim 1, the obviousness for combining of a shared reference region of Yildiz into Williamson is provided above. Prior art from Mori can further teach this limitation.
21 Mori teaches further comprising converting device dependent color space data on at least one shared reference region for the acquired live input image and the at least one shared reference region for the acquired target image to device independent color space data ([0014] reciting “a converting unit for converting data, defined by a color space independent of the image forming apparatus, of the data into first image data defined by a color space dependent of the image forming apparatus depending on the first environment light and then for converting the first image data into second image data defined by a color space independent of the image forming apparatus depending on the second environment light;”).
22 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Williamson in view of Yildiz) to incorporate the teachings of Mori to provide a clearer method that can provide a type of color space that is dependent and independent for the types of images provided from the teachings of Williamson in view of Yildiz. Doing so would determine the color difference of the images as stated by Mori ([Abstract] recited).
23 Regarding claim 4, Williamson in view of Yildiz and Mori teaches the method according to claim 3 (see claims 1 and 3 rejections above), live input image (Williamson; [0065] reciting “A method for determining the 3D structure of the human form or object in real-time. Any of a number of methods can be used. In order to control the cost of the systems, several methods have been developed which make use of the images from the cameras in order to determine 3D structure.”; [0257] reciting “Once the approximate 3D shape of objects are known, a number of methods can be used to create novel 3D views of the object using the input video. These methods fall into a broad class known as Image-Based Rendering (IBR).”)
24 Mori from claim 3 can further teach the limitation, specifically further comprising converting the transformed portion of the acquired live input image from device independent color space data back to device dependent color space data ([0042] reciting “Next, the image processing apparatus converts the obtained image data (Lab data) which is independent of the device correspondingly to first environment light to be corrected (light to be corrected) to obtain device-dependent image data (which is image data converted into color area display which can be reproduced by the printer; hereinafter referred to as printer RGB data).”).
25 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Williamson in view of Yildiz) to incorporate the teachings of Mori to provide a clearer method that can provide a transition from independent to dependent using the color data for the types of live images provided from the teachings of Williamson in view of Yildiz. Doing so would determine the color difference of the images as stated by Mori ([Abstract] recited).
26 Claim 10 has similar limitations as of claim 3, therefore it is rejected under the same rationale as claim 3.
27 Claim 11 has similar limitations as of claim 4, therefore it is rejected under the same rationale as claim 4.
28 Claim(s) 6-7 and 13-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Williamson et al. (US 20040104935 A1) in view of Yildiz et al. (US 10540812 B1) as of claim 1, further in view of Zhao et al. (US 20210049347 A1).
29 Regarding claim 6, Williamson in view of Yildiz teaches the method according to claim 1 (see claim 1 rejection above), further comprising automatically determining a selection of the live input image and a selection of the target image (Williamson; [0022] reciting “An automatic calibration program selects pair of targets from an image from the video camera. The selected pairs of targets are identified and the position of each target is calculated relative to the camera. The position of each target in a pair is then calculated in relation to each other. The positions of each target pair are added to a list of relative target transforms.”; [0065] reciting “A method for determining the 3D structure of the human form or object in real-time. Any of a number of methods can be used. In order to control the cost of the systems, several methods have been developed which make use of the images from the cameras in order to determine 3D structure.”; [0257] reciting “Once the approximate 3D shape of objects are known, a number of methods can be used to create novel 3D views of the object using the input video. These methods fall into a broad class known as Image-Based Rendering (IBR).”; [Abstract] reciting “A plurality of target markers are distributed within a virtual environment room where each target is distinct from all other targets in the virtual environment room and distinct from rotated versions of itself. An automatic calibration program selects pair of targets from an image from the video camera.”)
30 Williamson in view of Yildiz does not explicitly teach further comprising automatically determining a selection of the live input image and a selection of the target image to be acquired based on a feature point detection operation.
31 Zhao teaches further comprising automatically determining a selection of the live input image and a selection of the target image to be acquired based on a feature point detection operation ([0028] reciting “In an optional implementation, the reference image processing module is further configured to determine a first confidence level of the reference face image by using a first target determining model and the reference image feature; perform the operation of using the reference image feature as an input to a feature point positioning model, to obtain the target feature point location of the target area in the reference image in a case that the first confidence level reaches a first preset confidence level threshold…”; [0062] reciting “The current image may be an image acquired in real time by the terminal 110 through a built-in or externally connected image acquisition apparatus.”).
32 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Williamson in view of Yildiz) to incorporate the teachings of Zhao to provide a method that can select the live images taught by Williamson in view of Yildiz to be based on a feature point detection. Doing so would allow the methods to determine the image in which the face area is detected as the reference image in a case that the face area is detected in the obtained image as stated by Zhao ([0025] recited).
33 Regarding claim 7, Williamson in view of Yildiz and Zhao teaches the method according to claim 6 (see claims 1 and 6 rejections above), further comprising
34 Zhao from claim 6 can further teach this limitation, specifically determining the shared regions by extracting at least one feature point for the acquired live input image and at least one feature point for the acquired target image ([0034] reciting “In an optional implementation, the model training module is further configured to use a sample image pixel difference between the sample target areas in the any two frames of training sample images as model training input data of a second image feature extraction model; use an output of the second image feature extraction model as an input to the feature point location difference determining model, the output of the second image feature extraction model being the sample image feature difference between the sample target areas in the any two frames of training sample images...”; [0062] reciting “The current image may be an image acquired in real time by the terminal 110 through a built-in or externally connected image acquisition apparatus.”).
35 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Williamson in view of Yildiz) to incorporate the teachings of Zhao to provide a method for extracting a feature point from the input and target images provided by Williamson in view of Yildiz. Doing so would allow the methods to determine the image in which the face area is detected as the reference image in a case that the face area is detected in the obtained image as stated by Zhao ([0025] recited).
36 Claim 13 has similar limitations as of claim 6, therefore it is rejected under the same rationale as claim 6.
37 Claim 14 has similar limitations as of claim 7, therefore it is rejected under the same rationale as claim 7.
38 Claim(s) 16-17 and 21-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Williamson et al. (US 20040104935 A1) in view of Yildiz et al. (US 10540812 B1) and Gao et al. (US 20140250516 A1).
39 Regarding claim 16, Williamson teaches a method for relighting captured images in a virtual reality environment ([0158] reciting “A more attractive approach to fast 3D model construction is shape-from-silhouette. A number of cameras are placed around the subject. Each pixel in each camera is classified as either belonging to the subject (foreground) or the background. The resulting foreground mask is called a "silhouette". Each pixel in each camera collects light over a (very narrow) rectangular-based pyramid in 3D space, where the vertex of the pyramid is at the focal point of the camera and the pyramid extends infinitely away from this.”; [0180] reciting “Third, unless the bi-directional reflectance distribution function is uniform, the actual reflected light will vary at different camera vantage points.”; [0327] reciting “The detection algorithm thresholds the intensity of the full-color video image before doing the preliminary stages of target detection. This is done using a fixed brightness threshold…”), the method comprising: acquiring a live input image that includes an object from a camera ([0058] reciting “Virtual Viewpoint Remote Collaboration consists of a series of simulation booths equipped with multiple cameras observing the participants' actions. The video from these cameras is captured and processed in real-time to produce information about the three-dimensional structure of each participant. From this 3D information, Virtual Viewpoint technology is able to synthesize an infinite number of views from any viewpoint in the space, in real-time and on inexpensive mass-market PC hardware.”; [0065] reciting “A method for determining the 3D structure of the human form or object in real-time. Any of a number of methods can be used. In order to control the cost of the systems, several methods have been developed which make use of the images from the cameras in order to determine 3D structure.”; [0257] reciting “Once the approximate 3D shape of objects are known, a number of methods can be used to create novel 3D views of the object using the input video. These methods fall into a broad class known as Image-Based Rendering (IBR).”);
acquiring a target image ([Abstract] reciting “A plurality of target markers are distributed within a virtual environment room where each target is distinct from all other targets in the virtual environment room and distinct from rotated versions of itself. An automatic calibration program selects pair of targets from an image from the video camera.”; [0297] reciting “The image digitizer 1702 signal is used for detection of targets within the camera view 1703. The targets within the camera view are detected 1704.”; [0305] reciting “The pattern inside the target can be thought of as an image at some particular resolution.”) ;
acquired live input image ([0065] reciting “A method for determining the 3D structure of the human form or object in real-time. Any of a number of methods can be used. In order to control the cost of the systems, several methods have been developed which make use of the images from the cameras in order to determine 3D structure.”; [0257] reciting “Once the approximate 3D shape of objects are known, a number of methods can be used to create novel 3D views of the object using the input video. These methods fall into a broad class known as Image-Based Rendering (IBR).”)
;
40 Although Williamson could teach transforming at least a portion of the acquired live input image by applying the transform matrix to relight the live input image ([0097] reciting “This matrix takes the 3D homogeneous coordinate in space and converts it into an image-centered coordinate.”); outputting the transformed portion of the acquired live input image as a relit image ([0321] reciting “The target pair processing is repeated until all target pairs in the image are processed 2112. If the captured image has completed the processing stage 2113, then the optimal set of transforms is calculated 2114 and the set of targets and transforms are output 2115.”); and displaying, in a virtual environment, the relit image of the acquired live input image ([0197] reciting “For each frame, the augmented reality system identifies the transformation matrix relating marker and camera positions. This is passed to the virtual viewpoint server, together with the estimated camera calibration matrix. The server responds by returning a 374.times.288 pixel, 24 bit color image, and a range estimate associated with each pixel. This simulated view of the remote collaborator is then superimposed on the original image and displayed to the user.”), prior art from Yildiz can teach this limitation further. Williamson does not explicitly teach acquiring a target image representing a previously captured image of the object; calculating a covariance matrix of the at least one shared reference region for the acquired live input image and the at least one shared reference region for the acquired target image from device dependent color space data associated with the respective at least one shared reference regions; determining a transform matrix based on the calculated covariance matrices.
41 Yildiz teaches acquiring a target image representing a previously captured image of the object ([Page 11; Column 2, Lines 51-55] reciting “In some cases, the method may include detecting movement of the real-world light source; simulating a corresponding movement of the virtual light source; and updating an illumination of the overlay image according to the movement.”; [Page 12; Column 4, Lines 48-51] reciting “In some cases, an IR or NIR SLAM camera may be further configured to capture thermographic images of target objects, and to provide accurate non-contact temperature measurements of those objects.”; [Page 14; Column 8, Lines 45-50] reciting “The map may be maintained using predictions (e.g., when HMD 102 moves) and/or corrections (e.g., camera 108 observes landmarks in the environment that have been previously mapped). In other cases, a map of environment 100A may be obtained, at least in part, from cloud 104.”);
…transforming at least a portion of the acquired live input image by applying the transform matrix to relight the live input image ([Page 11; Column 2, Lines 18-27] reciting “The program instructions may further cause the IHS to transform a real-world location of the real-world light source to a virtual location in a coordinate system used by the xR application. Additionally, or alternatively, the program instructions may further cause the IHS to classify the real-world light source as: point, spot, directional, or area. To indicate the property, the program instructions may further cause the IHS to build a look-up table (LUT) of virtual light sources by type, position, color, intensity, and size.”; [Page 11; Column 2, Lines 51-59] reciting “In some cases, the method may include detecting movement of the real-world light source; simulating a corresponding movement of the virtual light source; and updating an illumination of the overlay image according to the movement. The method may also include detecting a change in intensity of the real-world light source; simulating a corresponding change of intensity of the virtual light source; and updating an illumination of the overlay image according to the change in intensity.”; [Page 16; Column 11; Lines 58-63] reciting “STAR module 408 may operate with xR application 401 to track the position or movement of light sources in real-time, or near real-time using SLAM landmarks. Moreover, STAR module 408 may be configured to provide a software service described in FIG. 5.”);
outputting the transformed portion of the acquired live input image as a relit image ([Page 15; Column 9, Lines 44-48] reciting “The observed data and annotations may then be used to generate one or more machine-learned algorithms that map inputs (e.g., observation data from a depth camera) to desired outputs (e.g., body-part indices for relevant pixels).”; ([Page 11; Column 2, Lines 18-27] reciting “The program instructions may further cause the IHS to transform a real-world location of the real-world light source to a virtual location in a coordinate system used by the xR application. Additionally, or alternatively, the program instructions may further cause the IHS to classify the real-world light source as: point, spot, directional, or area. To indicate the property, the program instructions may further cause the IHS to build a look-up table (LUT) of virtual light sources by type, position, color, intensity, and size.”; [Page 11; Column 2, Lines 51-59] reciting “In some cases, the method may include detecting movement of the real-world light source; simulating a corresponding movement of the virtual light source; and updating an illumination of the overlay image according to the movement. The method may also include detecting a change in intensity of the real-world light source; simulating a corresponding change of intensity of the virtual light source; and updating an illumination of the overlay image according to the change in intensity.”);
and displaying, in a virtual environment, the relit image of the acquired live input image ([Abstract] reciting “cause the IHS to: detect a real-world light source using a sensor mounted on a Head-Mounted Device (HMD); identify a property of the real-world light source; and indicate the property to a rendering engine during execution of an xR application, where the rendering engine is configured to render an image for display by the HMD based on the property.”; [Page 13; Column 5, Lines 19-26] reciting “…create artificial light sources in a virtual world's coordinates that correspond to the real-world light sources, and periodically update the light sources as part of the rendering of virtual objects to be displayed by HMD 102. These systems and methods may periodically update light sources during the course of execution of the xR application, in response to movement, brightness, or color changes, etc.”).
42 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Williamson) to incorporate the teachings of Yildiz to provide a clearer method regarding determining when the image is previously captured, finding a transformation matrix, as well as outputting and displaying the live input image while utilizing the relit methods taught by Williamson. Doing so would include certain types of tracking including inertial and optical tracking as stated Yildiz ([Page 12; Column 4, Lines 26-37] recited).
43 Williamson in view of Yildiz does not explicitly teach calculating a covariance matrix of the at least one shared reference region for the acquired live input image and the at least one shared reference region for the acquired target image from device dependent color space data associated with the respective at least one shared reference regions;
44 Gao teaches calculating a covariance matrix of the at least one shared reference region for the acquired live input image and the at least one shared reference region for the acquired target image from device dependent color space data associated with the respective at least one shared reference regions ([0090-0096] reciting “Calculating the second characteristic value of the input face region image includes the steps of: dividing the input face region image into three sub-images; generating corresponding dual samples respectively with respect to the three sub-images; decomposing the three sub-images into a first sample and a second sample respectively according to the dual samples corresponding to the three sub-images; constructing a covariance matrix respectively with respect to the first sample and the second sample of the three sub-images; determining the orthogonal normalized eigenvectors of the covariance matrix of the first sample and the orthogonal normalized eigenvectors of the covariance matrix of the second sample respectively; according to the eigenspace formed by the orthogonal normalized eigenvectors of the covariance matrix of the first sample and the eigenspace formed by the orthogonal normalized eigenvectors of the covariance matrix of the second sample, determining the projections of the first sample and the second sample in the eigenspace”);
determining a transform matrix based on the calculated covariance matrices ([0069] reciting “Therefore, the first eigenspace U.sub.e and the second eigenspace U.sub.o may be respectively obtained with respect to X.sub.e and X.sub.o based on the feature extraction algorithm, and then, the eigenvectors with high recognition precision and large variance are selected from the first eigenspace U.sub.e and the second eigenspace U.sub.o to constitute the eigenspace U of X. Then, by using U as a feature transformation matrix, features are extracted through V=AU.”).
45 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Williamson in view of Yildiz) to incorporate the teachings of Gao to provide a type of covariance matrix that was generated or calculated based on the image results from Williamson in view of Yildiz. Doing so would allow the ability of replacing or updating face images as stated by Gao ([0099] recited).
46 Regarding claim 17, Williamson in view of Yildiz and Gao teaches the method according to claim 16 (see claim 16 rejection above), and although Williamson could teach further comprising determining at least one shared reference region for the acquired live input image and at least one shared reference region for the acquired target image based on respective extracted feature points on the acquired live input image and acquired target image (Williamson; [0065] reciting “A method for determining the 3D structure of the human form or object in real-time. Any of a number of methods can be used. In order to control the cost of the systems, several methods have been developed which make use of the images from the cameras in order to determine 3D structure.”; [0257] reciting “Once the approximate 3D shape of objects are known, a number of methods can be used to create novel 3D views of the object using the input video. These methods fall into a broad class known as Image-Based Rendering (IBR).”; [0077] reciting “The LEDs are activated one at a time so that any video image of the plane will have a single bright spot in the image. By capturing 64 images from each camera, each LED is imaged once by each camera. By sequencing the LEDs in a known order, software can determine the precise 3D position of the LED. Finally, by elevating the plane to different heights, a set of points in 3 dimensions can be acquired.”) prior art from Yildiz from claim 16 can teach this limitation further.
47 Yildiz teaches further comprising determining at least one shared reference region for the acquired live input image and at least one shared reference region for the acquired target image based on respective extracted feature points on the acquired live input image and acquired target image ([Page 15; Column 10, Lines 33-36] reciting “In some cases, gaze tracking module 405 may be configured to identify a direction, extent, and/or speed of movement of the user's eyes in real-time, during execution of an xR application (e.g., a gaze vector).”; [Page 16; Column 11, Lines 22-25] reciting “Scanpath: a series of short fixations and saccades alternating before the eyes reach a target location on the screen (e.g., scanpath direction, duration, length and area covered);”; [Page 17; Column 14, Lines 40-45] reciting “At block 601, method 600 may begin calibration using images received from camera 108 mounted on HMD 102, for example. At block 602, method 600 may prompt user 101 wearing HMD 102 to rotate in position around the entire room, to cover an entire spherical area.”; [Page 17; Column 13, Lines 62-67] reciting “The ML engine learns what features correspond to what labels, which enables the classifier to predict the class of future light sources. Features that are extracted from the images may include: the outline of the light source, shape, light intensity, perceived direction/angle, light color, distance from the light source, etc.”).
48 As explained in the rejection of claim 16, the obviousness for combining of a shared reference region of Yildiz into Williamson and Gao is provided above, as well as Yildiz’s teachings of the extracted features from these regions, utilizing the input images taught by Williamson in view of Gao. Doing so would allow the distinguishing between point and directional lights as stated by Yildiz ([Page 17; Column 14, Lines 4-7] recited).
49 Claim 21 has similar limitations as of claim 16, therefore it is rejected under the same rationale as claim 16.
50 Claim 22 has similar limitations as of claim 17, therefore it is rejected under the same rationale as claim 17.
51 Claim(s) 19-20 and 24-25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Williamson et al. (US 20040104935 A1) in view of Yildiz et al. (US 10540812 B1) and Gao et al. (US 20140250516 A1) as of claim 16, further in view of Mori et al. (US 20100157341 A1).
52 Regarding claim 19, Williamson in view of Yildiz and Gao teaches the method according to claim 16 (see claim 16 rejection above), Yildiz from claim 16 teaches further comprising: on at least one shared reference region for the acquired live input image and the at least one shared reference region for the acquired target image . As explained in the rejection of claim 1, the obviousness for combining of a shared reference region of Yildiz into Williamson and Gao is provided above. Prior art from Mori can further teach this limitation.
53 Mori teaches further comprising converting device dependent color space data on at least one shared reference region for the acquired live input image and the at least one shared reference region for the acquired target image may to device independent color space data ([0014] reciting “a converting unit for converting data, defined by a color space independent of the image forming apparatus, of the data into first image data defined by a color space dependent of the image forming apparatus depending on the first environment light and then for converting the first image data into second image data defined by a color space independent of the image forming apparatus depending on the second environment light;”).
54 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Williamson in view of Yildiz and Gao) to incorporate the teachings of Mori to provide a clearer method that can provide a type of color space that is dependent and independent for the types of images provided from the teachings of Williamson in view of Yildiz and Gao. Doing so would determine the color difference of the images as stated by Mori ([Abstract] recited).
55 Regarding claim 20, Williamson in view of Gao and Mori teaches the method according to claim 19 (see claims 1 and 19 rejections above), acquired live input image (Williamson; [0065] reciting “A method for determining the 3D structure of the human form or object in real-time. Any of a number of methods can be used. In order to control the cost of the systems, several methods have been developed which make use of the images from the cameras in order to determine 3D structure.”; [0257] reciting “Once the approximate 3D shape of objects are known, a number of methods can be used to create novel 3D views of the object using the input video. These methods fall into a broad class known as Image-Based Rendering (IBR).”) .
56 Mori from claim 19 can further teach the limitation, specifically further comprising converting the transformed portion of the acquired live input image from device independent color space data back to device dependent color space data ([0042] reciting “Next, the image processing apparatus converts the obtained image data (Lab data) which is independent of the device correspondingly to first environment light to be corrected (light to be corrected) to obtain device-dependent image data (which is image data converted into color area display which can be reproduced by the printer; hereinafter referred to as printer RGB data).”).
57 It would have been obvious to one with ordinary skill before the effective filing date of the claimed invention, to have modified the method (taught by Williamson in view of Yildiz and Gao) to incorporate the teachings of Mori to provide a clearer method that can provide a transition from independent to dependent using the color data for the types of live images provided from the teachings of Williamson in view of Yildiz and Gao. Doing so would determine the color difference of the images as stated by Mori ([Abstract] recited).
58 Claim 24 has similar limitations as of claim 19, therefore it is rejected under the same rationale as claim 19.
59 Claim 25 has similar limitations as of claim 20, therefore it is rejected under the same rationale as claim 20.
Conclusion
60 Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
61 Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHNNY TRAN LE whose telephone number is (571)272-5680. The examiner can normally be reached Mon-Thu: 7:30am-5pm; First Fridays Off; Second Fridays: 7:30am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHNNY T LE/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights