DETAILED ACTION
This office action is in response to the application filed on 02/15/2024. Claims 1-20 are pending and are examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The reference(s) listed on the Information Disclosure Statement(s) submitted on 05/15/2024 and 07/28/2025 has/have been considered by the examiner (see attached PTO-1449).
Claim Objections
Claims 10 and 20 are objected to because of the following informalities:
In Claim 10, lines 1-2 recite, “the image” (i.e., lacking clear antecedent basis). For purposes of examination, the limitation will be reasonably interpreted as - - the color image - - . However, the Applicant is invited to amend this limitation in any similar way to overcome this objection.
In Claim 20, the last line recites, “the computing device” (i.e., lacking clear antecedent basis). For purposes of examination, the limitation will be reasonably interpreted as - - a computing device - - . However, the Applicant is invited to amend this limitation in any similar way to overcome this objection.
Examiner respectfully requests from Applicant verification and requires appropriate correction regarding these matters.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 1 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter.
In Claim 1, “One or more computer storage media” is recited. However, it appears that one of ordinary skill in the art could interpret a computer storage media as nothing more than encompassing a signal only, and thus, as a signal, per se. Furthermore, there is no language in the claim or Specification by which the claim elements can be made functional and statutory. A claim to a computer storage media that encompasses a signal covers a non-statutory embodiment (signal, per se), and thus is directed to non-statutory subject matter. See MPEP § 2106. As such, a person of ordinary skill in the art would interpret the limitations to mean merely a computer storage media encompassing a signal only, which is non-statutory.
Therefore, the examiner suggests amending the claim to recite - - One or more non-transitory computer storage media - - .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 7-11, 13-14, 17-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Luo, US Patent Application Publication No.: 2021/0357691 A1, hereby Luo, in view of Puri et al., US Patent Application Publication No.: 2022/0101047 A1, hereby Puri.
Luo discloses the invention substantially as claimed. Regarding Claim 1, Luo discloses one or more computer storage media comprising computer-executable instructions that when executed by a computing device performs a method of generating an augmented image, the method (Figs. 3-4, 13-14, and 15A; see also [0117], [0139], [0142], [0240]-[0241]) comprising:
“receiving at an image signal processor (ISP) raw sensor data from a sensor that is associated with a camera (Figs. 3-4, [0023], disclosing an ISP performing demosaicing on raw image sensor data; Fig. 15A, [0208]; see also Fig. 13-14);
generating an image at the ISP using the raw sensor data (Figs. 3-4 and 13-14, [0023], disclosing an ISP performing demosaicing on raw image sensor data; Fig. 15A, [0208]; see also Fig. 13);
communicating the image from the ISP to a neural processing unit (NPU), wherein the NPU includes a machine-learning (ML) model trained to make an . . . about an input image (Figs. 3-4 and 13-14, [0023], [0040], and [0199], disclosing a second part of a face recognition process that employs an NPU in the form of a deep learning accelerator, DLA, which performs an inference related task to face matching/identification/authentication using image data generated by the ISP; see also Fig. 15A, [0208]);
generating, at the NPU using the ML model, a ML . . . about the image (Figs. 3-4 and 13-14, [0023], [0040], and [0199]; see also Fig. 15A, [0208]);
associating the ML . . . with the image to form an augmented image (Figs. 3-4 and 13-14, [0142], disclosing metadata appended to images; see also [0023], [0040], and [0199]; see also Fig. 15A, [0208]); and
communicating the augmented image from the ISP to a computing component (Figs. 3-4 and 13-14, [0023], [0117], [0139], and [0142]; see also [0040], [0199]; see also Figs. 13-14 and 15A, [0208]).”
However, although Luo generally discloses the claimed inference, Puri does explicitly disclose the claimed “ML inference (see Puri, Figs. 1-2 and 5A-5B, [0061]-[0067], explicitly disclosing machine learning inference data; Fig. 9).”
Accordingly, before the effective filing date, it would have been obvious to one of ordinary skill in the art, having the teachings of Luo and Puri (hereby Luo-Puri), to modify the computer storage media of Luo to use the claimed ML inference as in Puri. The motivation for doing so would have been to create the advantage of accounting for any errors or unnatural artifacts that may be produced by the object mask generation, allowing the MLM to adequately depict an object (see Puri, Figs. 1-2 and 5A-5B, [0061]-[0067]; Fig. 9).
Regarding Claim 8, Luo-Puri discloses each and every feature of independent Claim 1, as outlined above, and further discloses a method of generating an augmented image (Luo, Figs. 3-4, 13-14, and 15A; see also [0117], [0139], [0142]) comprising:
“receiving, at an image signal processor (ISP), raw color sensor data from a color sensor that is associated with a first type of camera (Luo, Figs. 3-4, [0023], disclosing an ISP performing demosaicing on raw image sensor data; Fig. 15A, [0208]; see also Fig. 13-14);
receiving, at the image signal processor (ISP), raw second sensor data from a second sensor that is associated with a second type of camera, wherein the first type of camera and second type of camera are different (Luo, Figs. 3-4, [0023], disclosing an ISP performing demosaicing on raw image sensor data; Fig. 15A, [0112] (infrared), and [0208] (RGB); see also Fig. 13-14);
generating a color image at the ISP using the raw color sensor data (Luo, Figs. 3-4, [0023], disclosing an ISP performing demosaicing on raw image sensor data; Fig. 15A, [0112] (infrared), and [0208] (RGB); see also Fig. 13-14);
generating a second image at the ISP using the second sensor data (Luo, Figs. 3-4, [0023], disclosing an ISP performing demosaicing on raw image sensor data; Fig. 15A, [0112] (infrared), and [0208] (RGB); see also Fig. 13-14);
communicating the color image and the second image from the ISP to a neural processing unit (NPU), wherein the NPU includes a machine-learning (ML) model trained to make (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199]; see also Fig. 15A, [0112] (infrared), [0208] (color)) an inference (Puri, Figs. 1-2 and 5A-5B, [0061]-[0067]; Fig. 9) about an input image (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199], disclosing a second part of a face recognition process that employs an NPU in the form of a deep learning accelerator, DLA, which performs an inference related task to face matching/identification/authentication using image data generated by the ISP; see also Fig. 15A, [0112] (infrared), [0208] (color));
receiving a ML inference (Puri, Figs. 1-2 and 5A-5B, [0061]-[0067]; Fig. 9) about the color image from the NPU, wherein the NPU generated (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199]; see also Fig. 15A, [0112] (infrared), [0208] (color)) the ML inference (Puri, Figs. 1-2 and 5A-5B, [0061]-[0067]; Fig. 9) using the second image as input (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199], disclosing a second part of a face recognition process that employs an NPU in the form of a deep learning accelerator, DLA, which performs an inference related task to face matching/identification/authentication using image data generated by the ISP; see also Fig. 15A, [0112] (infrared), [0208] (color)); and
communicating the ML inference (Puri, Figs. 1-2 and 5A-5B, [0061]-[0067]; Fig. 9) from the ISP to a computing component (Luo, Figs. 3-4 and 13-14, [0023], [0117], [0139], and [0142]; see also [0040], [0199]; see also Figs. 13-14 and 15A, [0208]).”
The motivation that was utilized in Claim 1 applies equally as well here.
Regarding Claim 17, Luo-Puri discloses each and every feature of independent Claims 1 and 8, as outlined above, and further discloses a camera system (Luo, Figs. 3-4, 13-14, and 15A; see also [0117], [0139], [0142]) comprising:
“a color image sensor (Luo, Figs. 3-4, [0023], disclosing an ISP performing demosaicing on raw image sensor data; Fig. 15A, [0208]; see also Fig. 13-14);
an image signal processor (ISP) communicatively coupled to the color image sensor (Luo, Figs. 3-4, [0023], disclosing an ISP performing demosaicing on raw image sensor data; Fig. 15A, [0208]; see also Fig. 13-14);
a neural processing unit (NPU) communicatively coupled to the ISP, wherein the NPU includes a machine-learning (ML) model trained to make (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199]; see also Fig. 15A, [0112] (infrared), [0208] (color)) an inference (Puri, Figs. 1-2 and 5A-5B, [0061]-[0067]; Fig. 9) about an input image (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199], disclosing a second part of a face recognition process that employs an NPU in the form of a deep learning accelerator, DLA, which performs an inference related task to face matching/identification/authentication using image data generated by the ISP; see also Fig. 15A, [0112] (infrared), [0208] (color)); and
an output port configured to output an augmented image comprising an image generated by the ISP and a ML (Luo, Figs. 3-4 and 13-14, [0023], [0117], [0139], and [0142]; see also [0040], [0199]; see also Figs. 13-14 and 15A, [0208]) inference (Puri, Figs. 1-2 and 5A-5B, [0061]-[0067]; Fig. 9) generated by the ML model (Luo, Figs. 3-4 and 13-14, [0023], [0117], [0139], and [0142]; see also [0040], [0199]; see also Figs. 13-14 and 15A, [0208]).”
The motivation that was utilized in Claims 1 and 8 applies equally as well here.
Regarding Claims 2 and 10, Luo-Puri discloses:
“wherein the ML inference indicates a region of the image depicting (Puri, Figs. 1-2 and 5A-5B, [0061]-[0067]; Fig. 9) a face (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199], disclosing a second part of a face recognition process that employs an NPU in the form of a deep learning accelerator, DLA, which performs an inference related task to face matching/identification/authentication using image data generated by the ISP; see also Fig. 15A, [0112] (infrared), [0208] (color)).”
The motivation that was utilized in Claim 1 applies equally as well here.
Regarding Claims 3 and 9, Luo-Puri discloses:
“wherein the ML (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199], disclosing a second part of a face recognition process that employs an NPU in the form of a deep learning accelerator, DLA, which performs an inference related task to face matching/identification/authentication using image data generated by the ISP; see also Fig. 15A, [0112] (infrared), [0208] (color)) inference is a segment identification (Puri, Figs. 1-2 and 5A-5B, [0061]-[0067]; Fig. 9).”
The motivation that was utilized in Claim 1 applies equally as well here.
Regarding Claim 7, Luo-Puri discloses:
“wherein the augmented image is communicated over a bus conforming to a Mobile Industry Processor Interface (MIPI) Alliance standard or a USB connection (Luo, [0117]; (Figs. 3-4 and 13-14, [0023], [0139], and [0142]; see also [0040], [0199]; see also Figs. 13-14 and 15A, [0208])).”
Regarding Claim 11, Luo-Puri discloses:
“associating the ML (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199]; see also Fig. 15A, [0112] (infrared), [0208] (color), [0142], disclosing the DLA results in using both RGB and IR sensors and their use as metadata appended to the images) inference (Puri, Figs. 1-2 and 5A-5B, [0061]-[0067]; Fig. 9) with the color image to form an augmented image (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199], disclosing a second part of a face recognition process that employs an NPU in the form of a deep learning accelerator, DLA, which performs an inference related task to face matching/identification/authentication using image data generated by the ISP; see also Fig. 15A, [0112] (infrared), [0208] (color), [0142], disclosing the DLA results in using both RGB and IR sensors and their use as metadata appended to the images).”
The motivation that was utilized in Claim 1 applies equally as well here.
Regarding Claim 13, Luo-Puri discloses:
“associating the ML (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199]; see also Fig. 15A, [0112] (infrared), [0208] (color), [0142], disclosing the DLA results in using both RGB and IR sensors and their use as metadata appended to the images) inference (Puri, Figs. 1-2 and 5A-5B, [0061]-[0067]; Fig. 9) with the second image to form an augmented image (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199], disclosing a second part of a face recognition process that employs an NPU in the form of a deep learning accelerator, DLA, which performs an inference related task to face matching/identification/authentication using image data generated by the ISP; see also Fig. 15A, [0112] (infrared), [0208] (color), [0142], disclosing the DLA results in using both RGB and IR sensors and their use as metadata appended to the images).”
The motivation that was utilized in Claim 1 applies equally as well here.
Regarding Claims 14 and 18, Luo-Puri discloses:
“wherein the second type of camera is an infrared camera (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199]; see also Fig. 15A, [0112] (infrared), [0208] (color), [0142], disclosing the DLA results in using both RGB and IR sensors and their use as metadata appended to the images).”
Regarding Claim 20, Luo-Puri discloses:
“wherein the camera system is physically integrated with the computing device (Luo, Figs. 3-4 and 13-14, [0090]; see also Fig. 15A).”
Claim Rejections - 35 USC § 103
Claims 4-6, 12, 15-16, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Luo-Puri, and in further view of Kishore et al., WO 2022/241307 A1, hereby Kishore.
Regarding Claims 4, 12, and 19, Luo-Puri discloses:
“wherein the ML (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199]; see also Fig. 15A, [0112] (infrared), [0208] (color), [0142], disclosing the DLA results in using both RGB and IR sensors and their use as metadata appended to the images) inference (Puri, Figs. 1-2 and 5A-5B, [0061]-[0067]; Fig. 9) is associated with the image using . . . (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199], disclosing a second part of a face recognition process that employs an NPU in the form of a deep learning accelerator, DLA, which performs an inference related task to face matching/identification/authentication using image data generated by the ISP; see also Fig. 15A, [0112] (infrared), [0208] (color), [0142], disclosing the DLA results in using both RGB and IR sensors and their use as metadata appended to the images).”
The motivation that was utilized in Claim 1 applies equally as well here.
However, although Luo-Puri does not expressly disclose the claimed steganography, Kishore does expressly disclose the following:
“wherein the ML inference is associated with the image using steganography (Abstract, Figs. 1 and 10, page 30, lines 21-24, page 31, page 35, line 22 to page 36, line 22).”
Accordingly, before the effective filing date, it would have been obvious to one of ordinary skill in the art, having the teachings of Luo-Puri and Kishore (hereby Luo-Puri-Kishore), to modify the computer storage media, method of generating an augmented image, and camera system of Luo-Puri to use the claimed steganography as in Kishore. The motivation for doing so would have been to create the advantage of providing increased protection for users on various internet platforms (see Kishore, Abstract, Figs. 1 and 10, page 30, lines 21-24, page 31, page 35, line 22 to page 36, line 22).
Regarding Claims 5 and 15, Luo-Puri-Kishore discloses:
“extracting the ML (Luo, Figs. 3-4 and 13-14, [0023], [0040], and [0199]; see also Fig. 15A, [0112] (infrared), [0208] (color), [0142], disclosing the DLA results in using both RGB and IR sensors and their use as metadata appended to the images) inference (Puri, Figs. 1-2 and 5A-5B, [0061]-[0067]; Fig. 9) from the augmented image to generate a separate image and the ML inference (Kishore, Abstract, Fig. 1, and Fig. 10 (last two steps), page 30, lines 21-24, page 31, page 35, line 22 to page 36, line 22).”
The motivation that was utilized in Claims 4, 12, and 19 applies equally as well here.
Regarding Claims 6 and 16, Luo-Puri-Kishore discloses:
“communicating the ML (Luo, Figs. 3-4 and 13-14, [0023], [0117], [0139], [[0142]; [0040], and [0199]; see also Fig. 15A, [0112] (infrared), [0208] (color), [0142], disclosing the DLA results in using both RGB and IR sensors and their use as metadata appended to the images) inference (Puri, Figs. 1-2 and 5A-5B, [0061]-[0067]; Fig. 9) to a receiving application (Luo, Figs. 3-4 and 13-14, [0023], [0117], [0139], [[0142]; [0040], and [0199]; see also Fig. 15A, [0112] (infrared), [0208] (color), [0142], disclosing the DLA results in using both RGB and IR sensors and their use as metadata appended to the images) that uses the ML inference as input to a separate ML process (Kishore, Abstract, Fig. 1, and Fig. 10 (last two steps), page 30, lines 21-24, page 31, page 35, line 22 to page 36, line 22).”
The motivation that was utilized in Claims 5 and 15 applies equally as well here.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Examiner notes that multiple references cited disclosing increasing inference/authenticity accuracy in machine learning techniques. For example, the following references show similar features in the claims, although not relied upon: Mansata (US 2024/0054773 A1), Figs. 1-7.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATHLEEN M WALSH whose telephone number is (571)270-0423. The examiner can normally be reached M-F 8:00 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chris Kelley can be reached at (571) 272-7331. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATHLEEN M WALSH/Primary Examiner, Art Unit 2482