DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) is acknowledged.
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 06/05/2024 and 01/14/2025 have been made record of and considered by the examiner.
Claim Objections
Claim 5 is objected to because of the following informalities: The term “label each point cloud pixel” in claim 5 could lead to unnecessary ambiguity. For example, each point cloud pixel could be labeled with a color name, such as red, green, or blue, which finds no basis in the disclosure. Instead, the region-based labelling is bounded by color values. The disclosure limits labelling each point cloud pixel based on results from the ray tracing operation, being part of or within a region, a white point, or direct/indirect reflections from the second light source. See [0035-0036]: “For example, each point within the color point cloud may be labelled as "WP1" (e.g., being part of or within the first region, the first white point, the first light source (202), etc.), "WP2 direct" (e.g., being direct or immediate reflections from the second light source (204)), or "WP2 indirect" (e.g., being indirect or secondary reflections from the second light source (204)). At step (608), the processor identifies boundaries between the first region and the second region. These boundaries define the color value (such as R, G, and B values) used to determine whether a pixel is labeled as WP1 or WP2.” Appropriate clarification is respectfully requested.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3, 4, 6, 7, and 10 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
The term “wherein the binary mask is indicative of reflections of the first white point and the second white point” in claim 3 creates ambiguity regarding creation of the binary mask which renders the claim indefinite. It is unclear to the examiner whether the binary mask is (1) a single mask for reflections of both white points, or (2) a mask based on whether a ray from the second light source intercepts the first white point. Turning to the specification, [0033] states “At step (410), the processor generates a binary or alpha mask based on the ray-tracing operation. For example, each point of the surface map may be given a binary or alpha value based on whether a ray from the second light source (204) hits (i.e. intercepts) the respective point.” Appropriate correction is required.
The term “determine, for each pixel in the one of the image frames, a distance between a value of the pixel, the first tone, and the second tone” in claim 6 creates ambiguity which renders the claim indefinite. The term “a distance between a value of the pixel, the first tone, and the second tone” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. Further, the wording used in the claim only appears in claim-like clauses of the specification (see ¶81, EEE 6), and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
As it is currently phrased, the limitation could be referring to either (1) a single distance between the pixel, first tone, and second tone, (2) a distance between a value of the pixel and the first tone, and a distance between a value of the pixel and the second tone, or (3) something else. It is unclear to the examiner whether the limitation is referring to determination of a color proximity as seen in [0037]: “At step (610), the processor computes the color proximity of each pixel to each white point. For example, the value of each pixel labelled "WP2 direct" or "WP2 indirect" is compared to the value of the second white point and the value of the k-means boundary identified at step (608). The value of each pixel labelled "WP1" is compared to the value of the first white point and the value of the k-means boundary identified at step (608). At step (612), the processor generates a weight map for white point adjustment. For example, each pixel with a color proximity distance between the first white point and the k-means boundary that is less than a distance threshold (for example, 75%) receives no white point adjustment (e.g., a weight value of 0.0).” As seen in [0037], the specification appears to support comparing the value of the pixel to the value of the first or second tone, based on the label of the pixel and the value of the identified boundary. Appropriate correction is required.
The term “secondary image” in claim 7 creates ambiguity which renders the claim indefinite. The Examiner is unsure how the “secondary image” is defined in the absence of a first image. Further, the term “secondary image” only occurs in claim-like clauses at the end of the specification, which fail to clarify the term. Appropriate correction is required.
It is unclear if the replacement step of claim 7 is performed before/after determination of a color mapping function (of claim 1) or before/after application of the color mapping function (of claim 1). The specification (see [0046]) fails to clarify the matter. Further, the term “copy” is only mentioned in the claim-like clauses of the specification. ¶46 simply states: “the processor may identify content shown on the display device… the processor may determine that the content provided on the display device is stored in a server related to the video delivery pipeline (100)… processor may then retrieve the content [copy] from the server and replace the content.” Appropriate correction is required.
Regarding the limitation “subtract the second region from the one of the image frames to create a background image” in claim 10, the terms “subtract” and “create a background image” only occur in claim-like clauses at the end of the specification, which fail to clarify the limitation. Instead, the specification (¶53) discloses: “A source HDR image (1102) is provided to a first processor (1104). The first processor (1104) separates the source HDR image (1102) separated into a foreground layer and a modulation layer at step (1106).” The Examiner is unsure if the background of the claim corresponds to the modulation layer. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 5-10, and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Guan (US 2011/0298946 A1), in further view of Kadoi (US 2021/0274102 A1), and in further view of Morikawa (US 2021/0014465 A1).
Consider claims 1 and 14, Guan discloses a image delivery system/method for context-dependent color mapping, the image delivery system comprising (FIGs. 2 Process 200, 8, 9, ¶66, 84; an image processing apparatus):
a processor to perform post-production editing of image data including a plurality of image frames (FIG. 9 #24, ¶72, 84), the processor configured to:
identify a first region of one of the image frames (FIG. 10, ¶84; S204, “separation between a sunlit area and a shaded area is performed”, first region is sunlit area), the first region including a first white point having a first tone (¶57, 99; “a white portion of an image in the sunlit area is searched for and located to calculate gains, or Rg_B. Gg_B, and Bg_B.”);
identify a second region of the one of the image frames (FIG. 10, ¶84; S204, “separation between a sunlit area and a shaded area is performed”, second region is shaded area), the second region including a second white point having a second tone (¶57, 99; “A white portion of an image in the shaded area is searched for and located to calculate Rg_S. Gg_S, and Bg_S, which are gains for R, G, and B in the white portion in the shaded area.”);
determine a color mapping function based on the first tone and the second tone (¶85; S205, “image-Color correction is performed on each of the sunlit area and the shaded area separated at Step S204”);
apply the color mapping function to the second region (¶85; “appropriate automatic white balance control is performed on each of the sunlit area and the shaded area”); and
generate an output image for each of the plurality of image frames (¶47, 162; “The image output unit 304 outputs an image processed by the image-correction processing unit”).
While disclosing region-specific image correction, Guan does not explicitly disclose a video delivery system/method for context-dependent color mapping, the video delivery system comprising:
a processor to perform post-production editing of video data including a plurality of image frames, the processor configured to:
the first region including a first white point having a first tone,
the second region including a second white point having a second tone.
In related art, Kadoi discloses a video delivery system/method for context-dependent color mapping, the video delivery system (Kadoi FIG. 1A) comprising:
a processor to perform post-production editing of video data including a plurality of image frames, the processor configured to (Kadoi FIG. 1A #103, 105):
identify a first region of one of the image frames, the first region including a first white point having a first tone (Kadoi ¶113; “light source 1111 that illuminates the area of interest”);
identify a second region of the one of the image frames, the second region including a second white point having a second tone (Kadoi ¶113; “a large portion of the captured scene is illuminated by a light source 1112 having a color temperature different from that of the light source 1111 that illuminates the area of interest);
determine a color mapping function based on the first tone and the second tone (Kadoi ¶113; “a color temperature for obtaining a white balance coefficient suitable for applying to the entire image is calculated considering the color temperatures of the plurality of light sources that illuminate the captured scene 1113”);
apply the color mapping function to the second region (Kadoi ¶113-116; “suitable for applying to the entire image”); and
generate an output image for each of the plurality of image frames.
Guan acknowledges that “performing white balance control uniformly across an entire screen of the image can make either the sunlit area or the shaded area in the image to be unnatural in color (Guan ¶93).” Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the explicit region-specific color temperature calculations of Kadoi into the image delivery system of Guan to yield the predictable result of improved region-specific color reproduction in scenes containing multiple illuminants (Kadoi ¶112-116).
In related art, Morikawa supports:
a video delivery system/method for context-dependent color mapping, the video delivery system comprising (Morikawa Abstract, ¶26, 30-31):
a processor to perform post-production editing of video data including a plurality of image frames (Morikawa ¶51-55, 115-118; chroma suppression after white balance adjustment), the processor configured to:
identify a first region of one of the image frames (Morikawa ¶51; “the identification unit 402 identifies an image region with the chroma 503 in the image data 201 as a non-LED region with a primary subject (soccer player P) and a background”), the first region including a first white point having a first tone (Morikawa FIG. 5, #505, ¶55; chroma #503);
identify a second region of the one of the image frames (Morikawa ¶51; “the identification unit 402… identifies an image region in the image data 201 with the chroma 504 as an image region containing the LED billboard 103”), the second region including a second white point having a second tone (Morikawa FIG. 5, #505, ¶55; chroma #504);
determine a color mapping function based on the first tone and the second tone (Morikawa FIGs. 6-8, ¶55-61; chroma suppression process);
apply the color mapping function to the second region (Morikawa ¶55-61; chroma suppression process; ¶62; “the suppression unit 404 can execute the chroma suppression process on only the image region of the LED billboard 103”); and
generate an output image for each of the plurality of image frames (Morikawa FIGs. 11, 14 #S1109; ¶61; image with removed ambient light color).
Guan acknowledges that “performing white balance control uniformly across an entire screen of the image can make either the sunlit area or the shaded area in the image to be unnatural in color (Guan ¶93).” Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the techniques for chroma suppression of a light-emitting body included in the image data of Morikawa into the video delivery system of Guan, as modified by Kadoi, to further improve color rendering for specific regions. As shown by Morikawa, visibility of advertisements on an LED billboard may be improved by maintaining a white background (Morikawa ¶24). Morikawa additionally states that “By narrowing down the item to be subjected to the chroma suppression to image data of the specific light emitting body, it is possible to suppress the effect of chroma suppression on other image data such as the soccer player P, the spectators, and the field 102 that have already been subjected to white balance adjustment (Morikawa ¶116).”
Consider claim 5, Guan, as modified by Kadoi and Morikawa, discloses the claimed invention wherein the processor is further configured to:
create a three-dimensional color point cloud for the one of the image frames (Morikawa ¶92-93); and
label each point cloud pixel (Morikawa ¶51, 99; billboard presence or absence).
Consider claim 6, Guan, as modified by Kadoi and Morikawa, discloses the claimed invention wherein the processor is further configured to:
determine, for each pixel in the one of the image frames, a distance between a value of the pixel, the first tone, and the second tone (Kadoi ¶91, 141-149).
Consider claim 8, Guan, as modified by Kadoi and Morikawa, discloses the claimed invention wherein the processor is further configured to:
determine operating characteristics of a camera associated with the video data (Morikawa ¶26, 32, 127); and
determine operating characteristics of a backdrop display (Morikawa ¶24-26),
wherein the color mapping function is further based on the operating characteristics of the camera and the operating characteristics of the backdrop display (Morikawa ¶24-28, 78).
Consider claim 9, Guan, as modified by Kadoi and Morikawa, discloses the claimed invention wherein the camera associated with the video data is a camera used to capture the video data (Morikawa ¶26, 127), and
wherein at least a portion of the second region of the one of the image frames represents at least a portion of the backdrop display (Morikawa ¶24-28, 85-88).
Consider claim 10, Guan, as modified by Kadoi and Morikawa, discloses the claimed invention wherein the processor is further configured to:
subtract the second region from the one of the image frames to create a background image (Kadoi ¶84-87; Morikawa ¶24, 54-58);
identify a change in value for at least one pixel in the background image over subsequent image frames (Morikawa ¶54-58, 69); and
apply, in response to the change in value, the color mapping function to the at least one pixel in the background image (Guan ¶6; Morikawa ¶54-56, 69, 78).
Consider claim 15, Guan, as modified by Kadoi and Morikawa, discloses a non-transitory computer-readable medium storing instructions that, when executed by an electronic processor, cause the electronic processor to perform operations comprising the method of the claimed invention (Guan ¶75-77, 88; Kadoi ¶9).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Guan, in view of Kadoi and Morikawa, as applied to claims 1, 5-10, and 14-15 above, and further in view of Molyneaux (US 2021/0142581 A1).
Consider claim 2, Guan, as modified by Kadoi and Morikawa, discloses the claimed invention wherein the processor is further configured to:
receive a depth map associated with the one of the image frames (Kadoi ¶141-149); and
convert the depth map to a surface mesh (Morikawa ¶92-93).
In related art, Molyneaux further supports convert[ing] the depth map to a surface mesh (Molyneaux ¶102, 107, 112, 357-359).
Morikawa discloses rendering a three-dimensional model of a stadium, wherein each object of said stadium has three-dimensional location information. As stated by Morikawa, “By using 3-dimensional model data of the stadium 100, it is possible to detect an LED light source in the subject when performing still image or video capturing without the need to detect the flickering F.” Further, Molyneaux states that their 3D reconstruction may be stored in a suitable way, such as a point cloud or surface mesh (Molyneaux ¶359). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the surface mesh of Molyneaux into the system of Guan, as modified by Kadoi and Morikawa, to reconstruct and render objects using depth information, and store as a mesh (Molyneaux ¶46, N ¶93).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Guan, in view of Kadoi and Morikawa, as applied to claims 1, 5-10, and 14-15 above, and further in view of Kunkel (US 2015/0007243 A1).
Consider claim 7, Guan, as modified by Kadoi and Morikawa, discloses the claimed invention wherein the second region is a video display device, and wherein the processor is further configured to:
identify a secondary image within the second region (Kadoi ¶108);
receive a copy of the secondary image from a server (Morikawa ¶99).
In related art, Kunkel discloses: receive a copy of the secondary image from a server, wherein the copy of the secondary image received from the server has at least one of a higher resolution, a higher dynamic range, or a wider color gamut than the secondary image identified within the second region (Kunkel ¶51-55); and
replace the secondary image within the second region with the copy of the secondary image (Kunkel ¶51-55).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the replacement rendering of Kunkel into the video delivery system of Guan, as modified by Kadoi and Morikawa, to improve quality by providing post-production high definition/high dynamic range versions (Kunkel ¶55).
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Guan, in view of Kadoi and Morikawa, as applied to claims 1, 5-10, and 14-15 above, and further in view of Kitajima (US 2014/0147091 A1).
Consider claim 13, Guan, as modified by Kadoi and Morikawa, fails to specifically disclose:
store the color mapping function as metadata; and
transmit the metadata and the output image to an external device.
In related art, Kitajima discloses: store the color mapping function as metadata (Kitajima ¶72); and
transmit the metadata and the output image to an external device (Kitajima ¶84, 95-97, 112).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the storing of the color mapping function as metadata of Kitajima into the video delivery system of Guan, as modified by Kadoi and Morikawa, to improve color correction. As stated by Kitajima, “Performing provisional color grading on the set during image sensing in this manner makes it possible to confirm the appearance the image will ultimately have while sensing the image, and furthermore makes it possible to reduce the processing load of the final color grading carried out after image sensing and recording (Kitajima ¶4).”
Allowable Subject Matter
Claims 3-4 and 11-12 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten (1) in independent form including all of the limitations of the base claim and any intervening claims / (2) to overcome other applied rejections.
Relevant Prior Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 2017/0295372 A1 discloses a method of video coding using an image data correction mask.
JP2024098589A discloses a method to achieve appropriate color reproduction even when light from a display device is used as illumination light in photographing, using the display device as the background.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ASHLEY HYTREK whose telephone number is (703)756-4562. The examiner can normally be reached M-F 9:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Steve Koziol can be reached at (408)918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ASHLEY HYTREK/Examiner, Art Unit 2665
/Stephen R Koziol/Supervisory Patent Examiner, Art Unit 2665
Prosecution Insights