DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The listing of references in the specification is not a proper information disclosure statement. 37 CFR 1.98(b) requires a list of all patents, publications, or other information submitted for consideration by the Office, and MPEP § 609.04(a) states, "the list may not be incorporated into the specification but must be submitted in a separate paper." Therefore, unless the references have been cited by the examiner on form PTO-892, they have not been considered.
Specification
The disclosure is objected to because it contains an embedded hyperlink and/or other form of browser-executable code, see at least page 52 paragraph 0203 and page 53 paragraph 0206 of the instant specification. Applicant is required to delete the embedded hyperlink and/or other form of browser-executable code; references to websites should be limited to the top-level domain name without any prefix such as http:// or other browser-executable code. See MPEP § 608.01.
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Claim Objections
Claim 10 is objected to because of the following informalities: Lines 3 - 4 of claim 10 recite, in part, “user interface element; and processing a condition name” which appears to contain a grammatical error and/or a minor informality. The Examiner suggests amending the claim to --user interface element; [[and]] processing a condition name-- in order to improve the clarity and precision of the claim. Appropriate correction is required.
Claim 11 is objected to because of the following informalities: Line 1 of claim 11 recites, in part, “method skin condition search, the method” which appears to contain a grammatical error and/or a minor informality. The Examiner suggests amending the claim to --method for skin condition search, the method-- in order to improve the clarity and precision of the claim. Appropriate correction is required.
Claim 15 is objected to because of the following informalities: Line 6 of claim 15 recites, in part, “wherein each predicted condition classification is descriptive of a” which appears to contain a minor informality. The Examiner suggests amending the claim to --wherein each predicted condition classification of the plurality of predicted condition classifications is descriptive of a-- in order to improve the clarity and precision of the claim. Appropriate correction is required.
Claim 17 is objected to because of the following informalities: Line 11 of claim 17 recites, in part, “providing the one or more images to a medical condition classification model” which appears to contain inconsistent claim terminology and/or a minor informality. The Examiner suggests amending the claim to --providing the one or more images to a medical conditions classification model-- in order to maintain consistency with line 13 of claim 17 and to improve the clarity and precision of the claim. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 11 - 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 11 recites the limitation "the user" in line 9. There is insufficient antecedent basis for this limitation in the claim.
Claim 14 recites the limitation "the particular visual search result" in line 6. There is insufficient antecedent basis for this limitation in the claim.
Claim 17 recites the limitation "the particular candidate medical condition" in lines 18 - 19. There is insufficient antecedent basis for this limitation in the claim.
Claim 18 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention because it is unclear as to which one or more visual search results “the one or more visual search results” recited on line 18 is referencing. Is it referring to the “one or more visual search results” recited on lines 20 - 21 of claim 17 or the “one or more visual search results” recited on lines 2 - 3 of claim 18? For purposes of examination, the Examiner will treat the claim(s) as referencing a single same set of “one or more visual search results” and suggests amending line 2 of claim 18 to --processing the one or more images with [[a]] the search engine to determine the one or more visual search--.
Claim 19 recites the limitation "the ranking" in line 8. There is insufficient antecedent basis for this limitation in the claim.
Claims 12, 13, 15, 16 and 20 are also rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, due to being dependent upon a rejected base claim(s) but would be withdrawn from the rejection if their base claim(s) overcome the rejection.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 2, 5 - 7, 9 - 13, 15 and 16 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception, an abstract idea, without significantly more. The claims are directed towards generating candidate medical/skin conditions for images, which is an abstract idea.
The claims recite, at a high level of generality, processing the one or more images to determine the one or more images depict skin, processing the one or more images to generate an intent classification, wherein the intent classification indicates that the search query has a diagnostic search intent, and based on the intent classification, processing the one or more images to generate one or more predicted condition classifications, wherein the one or more predicted condition classifications are descriptive of one or more candidate medical/skin conditions determined to be potentially depicted in the one or more images.
The limitation of “processing… the one or more images… to determine the one or more images depict skin”, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind using observation, evaluation, judgment, and opinion but for the recitation of generic computer components. That is, other than reciting “a computing system comprising one or more processors” (see claim 11) and “a skin classification model” (see claim 11) nothing in the claim element precludes the step from practically being performed in the mind. The Examiner asserts that the claim(s) do not provide any details nor limit how the skin classification model operates or how it determines the one or more images depict skin, and the plain meaning of “determine” encompasses mental observations, evaluations, judgments and/or opinions, e.g., a user mentally deciding whether or not skin is depicted in an image. Under its broadest reasonable interpretation when read in light of the specification, the “determin[ing]” encompasses mental observations, evaluations, judgments and/or opinions that are practically performed in the human mind. For example, but for the recitation of the aforementioned generic computer components, the processing the one or more images to determine the one or more images depict skin encompasses a user observing an image and performing an evaluation, judgment and/or opinion by mentally deciding whether any skin is depicted in the image. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, with or without the use of a physical aid such as pen and paper, then it falls within the “Mental Processes” grouping of abstract ideas. See MPEP § 2106.04(a)(2)(III).
Similarly, the limitation of “processing the one or more images… to generate an intent classification, wherein the intent classification indicates that the search query has a diagnostic search intent”, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind using observation, evaluation, judgment, and opinion but for the recitation of generic computer components. That is, other than reciting “one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations” (see claim 1), “a computing system comprising one or more processors” (see claim 11) and “an intent classification model” nothing in the claim element precludes the step from practically being performed in the mind. The Examiner asserts that the claim(s) do not provide any details nor limit how the intent classification model operates or how it generates the intent classification, and the plain meaning of “generate” encompasses mental observations, evaluations, judgments and/or opinions, e.g., a user mentally deciding whether or not an image(s) has a diagnostic search intent. Under its broadest reasonable interpretation when read in light of the specification, the “generat[ing]” encompasses mental observations, evaluations, judgments and/or opinions that are practically performed in the human mind. For example, but for the recitation of the aforementioned generic computer components, the processing the one or more images to generate an intent classification indicating that the search query has a diagnostic search intent encompasses a user observing an image and performing an evaluation, judgment and/or opinion by mentally deciding whether anything depicted in the image should be looked at by a doctor, i.e. deciding that the image has a diagnostic search intent. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, with or without the use of a physical aid such as pen and paper, then it falls within the “Mental Processes” grouping of abstract ideas. See MPEP § 2106.04(a)(2)(III).
Relatedly, the limitation of “based on the intent classification; processing the one or more images… to generate one or more predicted condition classifications, wherein the one or more predicted condition classifications are descriptive of one or more candidate medical/skin conditions determined to be potentially depicted in the one or more images”, as drafted, is a process that, under its broadest reasonable interpretation, covers performance of the limitation in the mind using observation, evaluation, judgment, and opinion but for the recitation of generic computer components. That is, other than reciting “one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations” (see claim 1), “a computing system comprising one or more processors” (see claim 11), “the medical conditions classification model” (see claim 1) and “the dermatology conditions classification model” (see claim 11) nothing in the claim element precludes the step from practically being performed in the mind. The Examiner asserts that the claim(s) do not provide any details nor limit how the medical/dermatology conditions classification model operate or how they generates the one or more predicted condition classifications, and the plain meaning of “generate” encompasses mental observations, evaluations, judgments and/or opinions, e.g., a user mentally identifying one or more potential conditions depicted in an image. Under its broadest reasonable interpretation when read in light of the specification, the “generat[ing]” encompasses mental observations, evaluations, judgments and/or opinions that are practically performed in the human mind. For example, but for the recitation of the aforementioned generic computer components, the processing the one or more images to generate one or more predicted condition classifications encompasses a user observing an image and performing an evaluation, judgment and/or opinion to mentally decide on one or more potential conditions depicted in the image. If a claim limitation, under its broadest reasonable interpretation, covers performance of the limitation in the mind but for the recitation of generic computer components, with or without the use of a physical aid such as pen and paper, then it falls within the “Mental Processes” grouping of abstract ideas. See MPEP § 2106.04(a)(2)(III).
This judicial exception is not integrated into a practical application. In particular, the claims recite additional elements of: “one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations”, “obtaining a search query, wherein the search query comprises one or more images, wherein the one or more images depict a body part of a user”, “an intent classification model”, “providing the one or more images to a medical conditions classification model”, “providing medical condition information associated with the one or more candidate medical conditions”, “a computing system comprising one or more processors”, “a skin classification model”, “providing… the one or more images to a dermatology conditions classification model” and “providing… skin condition information associated with the one or more candidate skin conditions as an output.”
The limitations of “one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations” and “a computing system comprising one or more processors” are recited at a high level of generality such that they amount to no more than mere instructions to apply the exception using generic computer components. Furthermore, the claims as a whole merely describe how to generally “apply” the concept of generating candidate medical/skin conditions for images in a computer environment. Simply implementing the abstract idea on a generic computer is not a practical application of the abstract idea. See MPEP § 2106.05(f).
Further, the limitations of “obtaining a search query, wherein the search query comprises one or more images, wherein the one or more images depict a body part of a user”, “providing the one or more images to a medical conditions classification model”, “providing medical condition information associated with the one or more candidate medical conditions”, “providing… the one or more images to a dermatology conditions classification model” and “providing… skin condition information associated with the one or more candidate skin conditions as an output” are mere data gathering and output recited at a high level of generality, and thus are insignificant extra-solution activity. See MPEP § 2106.05(g). In addition, all uses of the recited judicial exception require such data gathering and output, and, as such, these limitations do not impose any meaningful limits on the claims. These limitations amount to necessary data gathering. See MPEP § 2106.05. Additionally, the elements of the aforementioned limitations amount to recording and transmitting digital images and/or information by use of conventional or generic technology in a nascent but well-known environment and are well-understood, routine, conventional activity. See MPEP § 2106.05(d).
Additionally, the limitations of “an intent classification model”, “a medical conditions classification model”, “a skin classification model” and “a dermatology conditions classification model” provide nothing more than mere instructions to implement an abstract idea on a generic computer. See MPEP § 2106.05(f). MPEP 2106.05(f) provides the following considerations for determining whether a claim simply recites a judicial exception with the words “apply it” (or an equivalent), such as mere instructions to implement an abstract idea on a computer: (1) whether the claim recites only the idea of a solution or outcome i.e., the claim fails to recite details of how a solution to a problem is accomplished; (2) whether the claim invokes computers or other machinery merely as a tool to perform an existing process; and (3) the particularity or generality of the application of the judicial exception. Moreover, the aforementioned models are used to generally apply the abstract idea without placing any limits on how the aforementioned models function. See MPEP 2106.05(f). Additionally, the recitations of “an intent classification model”, “a medical conditions classification model”, “a skin classification model” and “a dermatology conditions classification model” merely indicate a field of use or technological environment in which the judicial exception is performed. Although the additional elements of the aforementioned models limit the identified judicial exception of generating candidate medical/skin conditions for images, these types of limitations merely confine the use of the abstract idea to a particular technological environment (machine learning) and thus fail to add an inventive concept to the claims. See MPEP 2106.05(h).
Even when viewed in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Accordingly, the claims are directed to an abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements of: “one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations”, “obtaining a search query, wherein the search query comprises one or more images, wherein the one or more images depict a body part of a user”, “an intent classification model”, “providing the one or more images to a medical conditions classification model”, “providing medical condition information associated with the one or more candidate medical conditions”, “a computing system comprising one or more processors”, “a skin classification model”, “providing… the one or more images to a dermatology conditions classification model” and “providing… skin condition information associated with the one or more candidate skin conditions as an output” do not add a meaningful limitation to the abstract idea because they merely perform insignificant pre/post extrasolution activity, mere data gathering and output, and/or amount to no more than mere instructions to apply the abstract idea using generic computer components. Mere instructions to apply an exception using generic computer components cannot provide an inventive concept. The claims are not patent eligible.
In addition, with regards to dependent claims 2, 5 - 7, 9, 10, 12, 13, 15 and 16, the Examiner asserts that claims 2, 5 - 7, 9, 10, 12, 13, 15 and 16 are also directed to the abstract idea of generating candidate medical/skin conditions for images and dependent claims 2, 5 - 7, 9, 10, 12, 13, 15 and 16 merely further limit the abstract idea claimed in independent claims 1 and 11, for example by further identifying that only a portion of the one or more images is processed, by further identifying additional generic computer components, and/or by further identifying additional insignificant pre/post extrasolution activity that is performed. However, the Examiner asserts that a more detailed abstract idea remains an abstract idea and that none of the limitations of the dependent claims considered as an ordered combination provide eligibility because taken as a whole the claims merely instruct the practitioner to apply the abstract idea using generic computer components. The claims are not eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1 - 9 and 11 - 20 are rejected under 35 U.S.C. 103 as being unpatentable over Peng et al. U.S. Publication No. 2021/0019342 A1 in view of Lyman et al. U.S. Publication No. 2020/0160980 A1.
- With regards to claim 1, Peng et al. disclose a computing system for medical condition visual search, (Peng et al., Abstract, Figs. 1, 8 - 11 & 17, Pg. 1 ¶ 0001 and 0003, Pg. 3 ¶ 0040, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0061 - 0065) the system comprising: one or more processors; (Peng et al., Abstract, Fig. 8, Pg. 1 ¶ 0003, Pg. 2 ¶ 0016, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0061 and 0065, Pg. 8 ¶ 0088, Pg. 9 ¶ 0105 [“computer system”]) and one or more instructions that, when executed by the one or more processors, cause the computing system to perform operations, (Peng et al., Abstract, Fig. 8, Pg. 1 ¶ 0003, Pg. 2 ¶ 0016, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0061 and 0065, Pg. 8 ¶ 0088, Pg. 9 ¶ 0105 [“computer system”]) the operations comprising: obtaining a search query, wherein the search query comprises one or more images, wherein the one or more images depict a body part of a user; (Peng et al., Abstract, Figs. 1, 2, 9 - 12 & 17, Pg. 1 ¶ 0003 - 0004, Pg. 3 ¶ 0040 - 0042, Pg. 4 ¶ 0049 - 0051, Pg. 6 ¶ 0062 - 0065, Pg. 7 ¶ 0069 - 0071, Pg. 8 ¶ 0092 - 0099, Pg. 9 ¶ 0102 - 0105) wherein the search query has a diagnostic search intent; (Peng et al., Figs. 1, 2, 10, 11, 13 & 14, Pg. 1 ¶ 0008 and 0011 - 0012, Pg. 2 ¶ 0017 - 0020, Pg. 3 ¶ 0040 - 0044 and 0047, Pg. 4 ¶ 0050 - 0051, Pg. 5 ¶ 0054, Pg. 6 ¶ 0062 - Pg. 7 ¶ 0068, Pg. 7 ¶ 0070, Pg. 8 ¶ 0089 - 0091, 0093 and 0096 - 0099, Pg. 9 ¶ 0102 and 0110) providing the one or more images to a medical conditions classification model based on the diagnostic search intent; (Peng et al., Abstract, Figs. 6 - 11, Pg. 1 ¶ 0003 - 0005, Pg. 2 ¶ 0016 - 0020, Pg. 3 ¶ 0045 - 0047, Pg. 4 ¶ 0049 - 0050, Pg. 5 ¶ 0054 - 0059, Pg. 6 ¶ 0061 - 0063 and 0065, Pg. 7 ¶ 0083 - Pg. 8 ¶ 0086, Pg. 8 ¶ 0088) processing the one or more images with the medical conditions classification model to generate one or more predicted condition classifications, (Peng et al., Figs. 1, 2 & 10 - 12, Pg. 1 ¶ 0008 and 0012, Pg. 2 ¶ 0016 - 0020, Pg. 3 ¶ 0040 - 0044 and 0047, Pg. 4 ¶ 0050, Pg. 5 ¶ 0054, Pg. 6 ¶ 0062 - 0065, Pg. 7 ¶ 0069 - 0070, Pg. 8 ¶ 0088 - 0091 and 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0100) wherein the one or more predicted condition classifications are descriptive of one or more candidate medical conditions determined to be potentially depicted in the one or more images; (Peng et al., Figs. 1, 2 & 10 - 12, Pg. 1 ¶ 0008 and 0012, Pg. 3 ¶ 0040 - 0044 and 0047, Pg. 4 ¶ 0050, Pg. 6 ¶ 0063 - 0065, Pg. 7 ¶ 0070, Pg. 8 ¶ 0088 - 0091 and 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0100) and providing medical condition information associated with the one or more candidate medical conditions. (Peng et al., Figs. 1, 2 & 10 - 12, Pg. 1 ¶ 0008 and 0012, Pg. 3 ¶ 0040 - 0044 and 0047, Pg. 4 ¶ 0050, Pg. 6 ¶ 0063 - 0065, Pg. 7 ¶ 0070, Pg. 8 ¶ 0088 - 0091 and 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0100) Peng et al. fail to disclose explicitly one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations; processing the one or more images with an intent classification model to generate an intent classification, wherein the intent classification indicates that the search query has a diagnostic search intent, wherein the intent classification model was trained to determine a search intent of the user based on one or more features in an input image; and providing the images to the medical conditions classification model based on the intent classification. Pertaining to analogous art, Lyman et al. disclose a computing system for medical condition visual search, (Lyman et al., Figs. 1 - 3 & 15A - 16A, Pg. 1 ¶ 0034 - Pg. 2 ¶ 0039, Pg. 2 ¶ 0041, Pg. 4 ¶ 0047 - 0050, Pg. 4 ¶ 0053 - Pg. 5 ¶ 0056, Pg. 46 ¶ 0323 - 0327, Pg. 49 ¶ 0339 - 0342, Pg. 57 ¶ 0392, Pg. 57 ¶ 0397 - Pg. 58 ¶ 0398) the system comprising: one or more processors; (Lyman et al., Pg. 1 ¶ 0034 - Pg. 2 ¶ 0037, Pg. 4 ¶ 0050, Pg. 57 ¶ 0392, Pg. 57 ¶ 0397 - Pg. 58 ¶ 0398) and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, (Lyman et al., Pg. 1 ¶ 0034 - Pg. 2 ¶ 0037, Pg. 4 ¶ 0050, Pg. 57 ¶ 0392, Pg. 57 ¶ 0397 - Pg. 58 ¶ 0398) the operations comprising: obtaining a search query, wherein the search query comprises one or more images, wherein the one or more images depict a body part of a user; (Lyman et al., Figs. 1 - 3 & 15A - 16A, Pg. 14 ¶ 0108 - 0110, Pg. 46 ¶ 0323 - 0325, Pg. 46 ¶ 0327 - Pg. 47 ¶ 0332, Pg. 48 ¶ 0334 - 0336, Pg. 49 ¶ 0340 - 0342) processing the one or more images with an intent classification model to generate an intent classification, (Lyman et al., Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342 [“Each processing function 4008 can utilize a computer vision model trained to diagnose conditions of the corresponding type of image, for example, where each processing function 4008 is trained on a training set of images of the corresponding type as input data and on corresponding diagnosis data as output labels” and “multiple processing functions 4008 are performed in accordance with a hierarchy of processing functions 4008. For example, a first processing function 4008 trained to detect skin conditions is performed first on the captured image in response to determining the captured image is a picture of the skin. The diagnosis data generated by the first processing function can indicates a rash, and second processing functions 4008 trained to diagnose particular types of rashes can be selected to be performed next on the captured image to provide a more detailed diagnosis”]) wherein the intent classification indicates that the search query has a diagnostic search intent, (Lyman et al., Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342 [“Each processing function 4008 can utilize a computer vision model trained to diagnose conditions of the corresponding type of image, for example, where each processing function 4008 is trained on a training set of images of the corresponding type as input data and on corresponding diagnosis data as output labels” and “multiple processing functions 4008 are performed in accordance with a hierarchy of processing functions 4008. For example, a first processing function 4008 trained to detect skin conditions is performed first on the captured image in response to determining the captured image is a picture of the skin. The diagnosis data generated by the first processing function can indicates a rash, and second processing functions 4008 trained to diagnose particular types of rashes can be selected to be performed next on the captured image to provide a more detailed diagnosis”]) wherein the intent classification model was trained to determine a search intent of the user based on one or more features in an input image; (Lyman et al., Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342 [“Each processing function 4008 can utilize a computer vision model trained to diagnose conditions of the corresponding type of image, for example, where each processing function 4008 is trained on a training set of images of the corresponding type as input data and on corresponding diagnosis data as output labels”]) providing the one or more images to a medical conditions classification model based on the intent classification; (Lyman et al., Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342 [“multiple processing functions 4008 are performed in accordance with a hierarchy of processing functions 4008. For example, a first processing function 4008 trained to detect skin conditions is performed first on the captured image in response to determining the captured image is a picture of the skin. The diagnosis data generated by the first processing function can indicates a rash, and second processing functions 4008 trained to diagnose particular types of rashes can be selected to be performed next on the captured image to provide a more detailed diagnosis”]) processing the one or more images with the medical conditions classification model to generate one or more predicted condition classifications, (Lyman et al., Figs. 15A - 15C, Pg. 8 ¶ 0070 - 0075, Pg. 46 ¶ 0323 - 0326, Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342, Pg. 51 ¶ 0350 - 0353) wherein the one or more predicted condition classifications are descriptive of one or more candidate medical conditions determined to be potentially depicted in the one or more images; (Lyman et al., Figs. 15A - 15C, Pg. 8 ¶ 0070 - 0075, Pg. 46 ¶ 0323 - 0326, Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342, Pg. 51 ¶ 0350 - 0353) and providing medical condition information associated with the one or more candidate medical conditions. (Lyman et al., Figs. 15A - 15C, Pg. 8 ¶ 0070 - 0075, Pg. 46 ¶ 0323 - 0326, Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342, Pg. 51 ¶ 0350 - 0353) Peng et al. and Lyman et al. are combinable because they are both directed towards medical image processing systems that process dermatology images depicting skin to generate predicted diagnosis data for the dermatology images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Peng et al. with the teachings of Lyman et al. A first modification would have been prompted in order to enhance the base device of Peng et al. with the well-known and applicable technique Lyman et al. applied to a comparable device. Utilizing one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations, as taught by Lyman et al., to implement operations of the base device of Peng et al. would enhance the base device of Peng et al. by ensuring that its operations are carried out accurately and efficiently at high-computational speed on computer architecture, by facilitating widespread distribution of the base device of Peng et al. to millions of potential end-users with access to a computer and by simplifying the process of making revisions, modifications and/or updates to the operations of the base device of Peng et al. A second modification would have been prompted in order to enhance the base device of Peng et al. with the well-known and applicable technique Lyman et al. applied to a comparable device. Processing the one or more images with an intent classification model to generate an intent classification and providing the one or more images to the medical conditions classification model based on the intent classification, as taught by Lyman et al., would enhance the base device of Peng et al. by reducing the total number of image processing operations it performs since only search query images determined to have a diagnostic search intent would undergo processing by the medical conditions classification model thereby eliminating unnecessary image processing operations from being performed by the medical conditions classification model on search query images lacking diagnostic search intent and improving the overall computational efficiency of the base device of Peng et al. This combination could be completed according to well-known techniques in the art and would likely yield predictable results, in that one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause a computing system to perform operations would be utilized to implement the base device of Peng et al. so as to ensure that its operations are carried out accurately and efficiently at high-computational speed on computer architecture, to facilitate widespread distribution of the base device of Peng et al. to millions of potential end-users with access to a computer and to simplify the process of making revisions, modifications and/or updates to the operations of the base device of Peng et al. and in that the one or more images would be processed with an intent classification model to generate an intent classification and provided to the medical conditions classification model based on the intent classification so as to help ensure that unnecessary image processing operations are not performed by the medical conditions classification model on search query images lacking a diagnostic search intent and thereby improve the overall computational efficiency of the base device of Peng et al. Therefore, it would have been obvious to combine Peng et al. with Lyman et al. to obtain the invention as specified in claim 1.
- With regards to claim 2, Peng et al. in view of Lyman et al. disclose the system of claim 1. Peng et al. fail to disclose explicitly processing the one or more images with a skin classification model to determine the one or more images depict skin, wherein the skin classification model was trained to determine whether an input image depicts skin; and providing the one or more images to the intent classification model based on the one or more images depicting skin. Pertaining to analogous art, Lyman et al. disclose processing the one or more images with a skin classification model to determine the one or more images depict skin, (Lyman et al., Pg. 46 ¶ 0323 - 0327, Pg. 47 ¶ 0329 - 0331, Pg. 49 ¶ 0341 - 0343) wherein the skin classification model was trained to determine whether an input image depicts skin; (Lyman et al., Pg. 46 ¶ 0323 - 0327, Pg. 47 ¶ 0329 - 0331, Pg. 49 ¶ 0341 - 0343) and providing the one or more images to the intent classification model based on the one or more images depicting skin. (Lyman et al., Pg. 47 ¶ 0329 - 0331, Pg. 49 ¶ 0342 [“multiple processing functions 4008 are performed in accordance with a hierarchy of processing functions 4008. For example, a first processing function 4008 trained to detect skin conditions is performed first on the captured image in response to determining the captured image is a picture of the skin. The diagnosis data generated by the first processing function can indicates a rash, and second processing functions 4008 trained to diagnose particular types of rashes can be selected to be performed next on the captured image to provide a more detailed diagnosis.”])
- With regards to claim 3, Peng et al. in view of Lyman et al. disclose the system of claim 1, wherein the operations further comprise: obtaining the medical condition information associated with the one or more candidate medical conditions from a curated medical information database, (Peng et al., Figs. 3, 8 & 9, Pg. 1 ¶ 0012 - 0013, Pg. 3 ¶ 0046 - 0047, Pg. 4 ¶ 0051, Pg. 6 ¶ 0062 - 0065) wherein the medical condition information comprises a medical condition name and one or more condition images, (Peng et al., Figs. 1, 2 & 10 - 12, Pg. 1 ¶ 0006 - 0008 and 0011 - 0013, Pg. 3 ¶ 0040 - 0044 and 0046 - 0047, Pg. 4 ¶ 0050 - 0051, Pg. 6 ¶ 0062 - 0065, Pg. 7 ¶ 0069 - 0070, Pg. 8 ¶ 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102) wherein the one or more condition images depict an example of the respective candidate medical condition. (Peng et al., Figs. 1, 2 & 10 - 12, Pg. 1 ¶ 0006 - 0008 and 0011 - 0013, Pg. 3 ¶ 0040 - 0044 and 0046 - 0047, Pg. 4 ¶ 0050 - 0051, Pg. 6 ¶ 0062 - 0065, Pg. 7 ¶ 0069 - 0070, Pg. 8 ¶ 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102)
- With regards to claim 4, Peng et al. in view of Lyman et al. disclose the system of claim 3, wherein the one or more condition images are obtained from a medical condition image database, (Peng et al., Figs. 3, 8 & 9, Pg. 1 ¶ 0012 - 0013, Pg. 3 ¶ 0046 - 0047, Pg. 4 ¶ 0051, Pg. 6 ¶ 0062 - 0065) wherein the medical condition image database comprises a plurality of medical condition images selected by one or more medical professionals. (Peng et al., Figs. 3, 8 & 9, Pg. 1 ¶ 0012 - 0013, Pg. 2 ¶ 0025, Pg. 3 ¶ 0046 - 0047, Pg. 4 ¶ 0051, Pg. 6 ¶ 0062 - Pg. 7 ¶ 0066, Pg. 7 ¶ 0069 - 0070, Pg. 8 ¶ 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102)
- With regards to claim 5, Peng et al. in view of Lyman et al. disclose the system of claim 1, wherein the operations further comprise: processing the one or more images to determine a region of interest; (Peng et al., Figs. 1 & 2, Pg. 1 ¶ 0006, Pg. 2 ¶ 0017 - 0018, Pg. 3 ¶ 0042 - 0043, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0062 and 0065, Pg. 8 ¶ 0089 - 0090 and 0099, Pg. 9 ¶ 0104 - 0106) cropping the one or more images to generate one or more cropped images based on the region of interest; (Peng et al., Figs. 1 & 2, Pg. 1 ¶ 0006, Pg. 2 ¶ 0017 - 0018, Pg. 3 ¶ 0042 - 0043, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0062 and 0065, Pg. 8 ¶ 0089 - 0090 and 0099, Pg. 9 ¶ 0104 - 0106) and wherein the one or more cropped images are processed with the medical conditions classification model. (Peng et al., Pg. 1 ¶ 0006, Pg. 3 ¶ 0042 - 0043, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0062, Pg. 8 ¶ 0099, Pg. 9 ¶ 0104 - 0105) Peng et al. fail to disclose explicitly wherein the one or more cropped images are processed with the intent classification model. Pertaining to analogous art, Lyman et al. disclose processing the one or more images to determine a region of interest; (Lyman et al., Pg. 20 ¶ 0141, Pg. 21 ¶ 0143 - 0145, Pg. 46 ¶ 0325, Pg. 47 ¶ 0329 - 0331, Pg. 49 ¶ 0341 - 0343, Pg. 51 ¶ 0353 - 0355, Pg. 52 ¶ 0361 - Pg. 53 ¶ 0362, Pg. 54 ¶ 0375) cropping the one or more images to generate one or more cropped images based on the region of interest; (Lyman et al., Pg. 20 ¶ 0141, Pg. 21 ¶ 0143 - 0145, Pg. 46 ¶ 0325, Pg. 47 ¶ 0329 - 0331, Pg. 49 ¶ 0341 - 0343, Pg. 51 ¶ 0353 - 0355, Pg. 52 ¶ 0361 - Pg. 53 ¶ 0362, Pg. 54 ¶ 0375) and wherein the one or more cropped images are processed with the intent classification model and the medical conditions classification model. (Lyman et al., Pg. 20 ¶ 0140 - Pg. 21 ¶ 0145, Pg. 46 ¶ 0325, Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0340 - 0342, Pg. 51 ¶ 0353 [“the captured image is pre-processed with filters to change brightness, contrast, orientation, and/or to crop the image before the image classifier function 4007 and/or processing functions 4008 is applied.”])
- With regards to claim 6, Peng et al. in view of Lyman et al. disclose the system of claim 1, wherein the operations further comprise: processing the one or more images to determine a region of interest; (Peng et al., Figs. 1 & 2, Pg. 1 ¶ 0006, Pg. 2 ¶ 0017 - 0018, Pg. 3 ¶ 0042 - 0043, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0062 and 0065, Pg. 8 ¶ 0089 - 0090 and 0099, Pg. 9 ¶ 0104 - 0106) generating an annotated image based on the one or more images and the region of interest, (Peng et al., Figs. 1, 2 & 10, Pg. 1 ¶ 0006 and 0011 - 0012, Pg. 2 ¶ 0017 - 0020, Pg. 3 ¶ 0042 - 0044 and 0047, Pg. 4 ¶ 0050, Pg. 5 ¶ 0054, Pg. 6 ¶ 0062 - 0065, Pg. 7 ¶ 0070, Pg. 8 ¶ 0089 - 0092 and 0096, Pg. 9 ¶ 0105 - 0110) wherein the annotated image comprises the one or more images with one or more indicators, (Peng et al., Figs. 1 & 2, Pg. 1 ¶ 0006 and 0011 - 0012, Pg. 2 ¶ 0017 - 0020, Pg. 3 ¶ 0042 - 0044, Pg. 4 ¶ 0050, Pg. 5 ¶ 0054, Pg. 6 ¶ 0062 - 0065, Pg. 7 ¶ 0070, Pg. 8 ¶ 0089 - 0092 and 0096, Pg. 9 ¶ 0105 - 0110) wherein the one or more indicators indicate a location of the region of interest in the one or more images; (Peng et al., Figs. 1 & 2, Pg. 1 ¶ 0006, Pg. 2 ¶ 0017 - 0020, Pg. 3 ¶ 0042 - 0043, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0062 - 0065, Pg. 8 ¶ 0089 - 0092, 0096 and 0099, Pg. 9 ¶ 0105 - 0110) and providing the annotated image for display with the medical condition information as an output. (Peng et al., Figs.1, 2, 10, 11 & 17, Pg. 1 ¶ 0003, 0006 - 0009 and 0011 - 0012, Pg. 2 ¶ 0017 - 0022, Pg. 3 ¶ 0042 - 0044, Pg. 4 ¶ 0049 - 0050, Pg. 5 ¶ 0054, Pg. 6 ¶ 0063 - 0065, Pg. 7 ¶ 0070, Pg. 8 ¶ 0089 - 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0100, Pg. 9 ¶ 0104 - 0110) In addition, analogous art Lyman et al. disclose processing the one or more images to determine a region of interest; (Lyman et al., Pg. 20 ¶ 0141, Pg. 21 ¶ 0143 - 0145, Pg. 46 ¶ 0325, Pg. 47 ¶ 0329 - 0331, Pg. 49 ¶ 0341 - 0343, Pg. 51 ¶ 0353 - 0355, Pg. 52 ¶ 0361 - Pg. 53 ¶ 0362, Pg. 54 ¶ 0375) generating an annotated image based on the one or more images and the region of interest, (Lyman et al., Fig. 16D, Pg. 51 ¶ 0352 - 0353, Pg. 54 ¶ 0375) wherein the annotated image comprises the one or more images with one or more indicators, (Lyman et al., Fig. 16D, Pg. 51 ¶ 0352 - 0353, Pg. 54 ¶ 0375) wherein the one or more indicators indicate a location of the region of interest in the one or more images; (Lyman et al., Fig. 16D, Pg. 51 ¶ 0352 - 0353, Pg. 54 ¶ 0375) and providing the annotated image for display with the medical condition information as an output. (Lyman et al., Fig. 16D, Pg. 51 ¶ 0352 - 0353, Pg. 54 ¶ 0375)
- With regards to claim 7, Peng et al. in view of Lyman et al. disclose the system of claim 1, wherein providing medical condition information associated with the one or more candidate medical conditions comprises: providing the medical condition information for display in a search results interface. (Peng et al., Figs. 1, 2, 10, 11 & 17, Pg. 1 ¶ 0006 - 0008 and 0011 - 0012, Pg. 2 ¶ 0017 - 0022, Pg. 3 ¶ 0041 - 0044, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0061 - 0065, Pg. 7 ¶ 0070, Pg. 8 ¶ 0086 - 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0100, Pg. 9 ¶ 0102 - 0110)
- With regards to claim 8, Peng et al. in view of Lyman et al. disclose the system of claim 7, wherein the search results interface comprises: a first panel comprising the medical condition information; (Peng et al., Figs. 1, 2, 10 & 11, Pg. 3 ¶ 0043 - 0044, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0063 - 0065, Pg. 8 ¶ 0094 - 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0100) and a second panel comprising a plurality of visual search results, (Peng et al., Figs. 1, 2, 10, 11 & 17, Pg. 1 ¶ 0006 - 0008 and 0011, Pg. 2 ¶ 0022 - 0024 and 0032 - 0033, Pg. 3 ¶ 0043 - 0044, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0061 - 0065, Pg. 8 ¶ 0086 - 0087 and 0094 - 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0100, Pg. 9 ¶ 0102 and 0110) wherein the plurality of visual search results are determined based on a determined visual similarity with the one or more images. (Peng et al., Abstract, Figs. 6 & 17, Pg. 1 ¶ 0003 and 0005 - 0007, Pg. 4 ¶ 0048 - 0050, Pg. 5 ¶ 0052 - 0055 and 0059, Pg. 6 ¶ 0061)
- With regards to claim 9, Peng et al. in view of Lyman et al. disclose the system of claim 7, wherein the search results interface comprises: a selectable user interface element, wherein the selectable user interface element is associated with a particular medical condition of the one or more candidate medical conditions, (Peng et al., Figs. 1, 10 & 17, Pg. 1 ¶ 0009, Pg. 4 ¶ 0049, Pg. 6 ¶ 0063 - 0064, Pg. 8 ¶ 0096 and 0099, Pg. 9 ¶ 0110) and wherein the selectable user interface element is provided adjacent to the medical condition information. (Peng et al., Figs. 1, 10 & 17, Pg. 1 ¶ 0008 - 0012, Pg. 4 ¶ 0048 - 0049, Pg. 6 ¶ 0061 - 0064, Pg. 8 ¶ 0086 - 0087, 0094 - 0096 and 0099)
- With regards to claim 11, Peng et al. disclose a computer-implemented method skin condition search, (Peng et al., Abstract, Figs. 1, 8 - 11 & 17, Pg. 1 ¶ 0001 and 0003 - 0004, Pg. 3 ¶ 0040 - 0044, Pg. 4 ¶ 0049 - 0051, Pg. 6 ¶ 0061 - 0065, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102, Pg. 9 ¶ 0110) the method comprising: obtaining, by a computing system comprising one or more processors, (Peng et al., Abstract, Fig. 8, Pg. 1 ¶ 0003, Pg. 2 ¶ 0016, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0061 and 0065, Pg. 8 ¶ 0088, Pg. 9 ¶ 0105 [“computer system”]) a search query, wherein the search query comprises one or more images; (Peng et al., Abstract, Figs. 1, 2, 9 - 12 & 17, Pg. 1 ¶ 0003 - 0004, Pg. 3 ¶ 0040 - 0042, Pg. 4 ¶ 0049 - 0051, Pg. 6 ¶ 0062 - 0065, Pg. 7 ¶ 0069 - 0071, Pg. 8 ¶ 0092 - 0099, Pg. 9 ¶ 0102 - 0105) wherein the search query has a diagnostic search intent; (Peng et al., Figs. 1, 2, 10, 11, 13 & 14, Pg. 1 ¶ 0008 and 0011 - 0012, Pg. 2 ¶ 0017 - 0020, Pg. 3 ¶ 0040 - 0044 and 0047, Pg. 4 ¶ 0050 - 0051, Pg. 5 ¶ 0054, Pg. 6 ¶ 0062 - Pg. 7 ¶ 0068, Pg. 7 ¶ 0070, Pg. 8 ¶ 0089 - 0091, 0093 and 0096 - 0099, Pg. 9 ¶ 0102 and 0110) providing, by the computing system, (Peng et al., Abstract, Fig. 8, Pg. 1 ¶ 0003, Pg. 2 ¶ 0016, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0061 and 0065, Pg. 8 ¶ 0088, Pg. 9 ¶ 0105 [“computer system”]) the one or more images to a dermatology conditions classification model based on the diagnostic search intent; (Peng et al., Abstract, Figs. 6 - 11, Pg. 1 ¶ 0003 - 0005, Pg. 2 ¶ 0016 - 0020, Pg. 3 ¶ 0045 - 0047, Pg. 4 ¶ 0049 - 0050, Pg. 5 ¶ 0054 - 0059, Pg. 6 ¶ 0061 - 0063 and 0065, Pg. 7 ¶ 0083 - Pg. 8 ¶ 0086, Pg. 8 ¶ 0088) processing, by the computing system, (Peng et al., Abstract, Fig. 8, Pg. 1 ¶ 0003, Pg. 2 ¶ 0016, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0061 and 0065, Pg. 8 ¶ 0088, Pg. 9 ¶ 0105 [“computer system”]) the one or more images with the dermatology conditions classification model to generate one or more predicted condition classifications, (Peng et al., Figs. 1, 2 & 10 - 12, Pg. 1 ¶ 0008 and 0012, Pg. 2 ¶ 0016 - 0020, Pg. 3 ¶ 0040 - 0044 and 0047, Pg. 4 ¶ 0050, Pg. 5 ¶ 0054, Pg. 6 ¶ 0062 - 0065, Pg. 7 ¶ 0069 - 0070, Pg. 8 ¶ 0088 - 0091 and 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0100) wherein the one or more predicted condition classifications are descriptive of one or more candidate skin conditions; (Peng et al., Figs. 1, 2 & 10 - 12, Pg. 1 ¶ 0004, 0008 and 0012, Pg. 3 ¶ 0040 - 0044 and 0047, Pg. 4 ¶ 0050 - 0051, Pg. 6 ¶ 0062 - 0065, Pg. 8 ¶ 0088 - 0091 and 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102) and providing, by the computing system, (Peng et al., Abstract, Fig. 8, Pg. 1 ¶ 0003, Pg. 2 ¶ 0016, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0061 and 0065, Pg. 8 ¶ 0088, Pg. 9 ¶ 0105 [“computer system”]) skin condition information associated with the one or more candidate skin conditions as an output. (Peng et al., Figs. 1, 2 & 10 - 12, Pg. 1 ¶ 0008 and 0012, Pg. 3 ¶ 0040 - 0044 and 0047, Pg. 4 ¶ 0050 - 0051, Pg. 6 ¶ 0062 - 0065, Pg. 7 ¶ 0070, Pg. 8 ¶ 0088 - 0091 and 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102) Peng et al. fail to disclose explicitly processing the one or more images with a skin classification model to determine the one or more images depict skin; processing the one or more images with an intent classification model to generate an intent classification, wherein the intent classification indicates that the search query has a diagnostic search intent, wherein the intent classification model was trained to determine a search intent of the user based on one or more features in an input image; and providing the images to the dermatology conditions classification model based on the intent classification. Pertaining to analogous art, Lyman et al. disclose a computer-implemented method skin condition search, (Lyman et al., Figs. 1 - 3 & 15A - 16A, Pg. 1 ¶ 0034 - Pg. 2 ¶ 0039, Pg. 2 ¶ 0041, Pg. 4 ¶ 0047 - 0050, Pg. 4 ¶ 0053 - Pg. 5 ¶ 0056, Pg. 46 ¶ 0323 - 0327, Pg. 49 ¶ 0339 - 0342, Pg. 57 ¶ 0392, Pg. 57 ¶ 0397 - Pg. 58 ¶ 0398) the method comprising: obtaining, by a computing system comprising one or more processors, (Lyman et al., Pg. 1 ¶ 0034 - Pg. 2 ¶ 0037, Pg. 4 ¶ 0050, Pg. 57 ¶ 0392, Pg. 57 ¶ 0397 - Pg. 58 ¶ 0398) a search query, wherein the search query comprises one or more images; (Lyman et al., Figs. 1 - 3 & 15A - 16A, Pg. 14 ¶ 0108 - 0110, Pg. 46 ¶ 0323 - 0325, Pg. 46 ¶ 0327 - Pg. 47 ¶ 0332, Pg. 48 ¶ 0334 - 0336, Pg. 49 ¶ 0340 - 0342) processing, by the computing system, (Lyman et al., Pg. 1 ¶ 0034 - Pg. 2 ¶ 0037, Pg. 4 ¶ 0050, Pg. 57 ¶ 0392, Pg. 57 ¶ 0397 - Pg. 58 ¶ 0398) the one or more images with a skin classification model to determine the one or more images depict skin; (Lyman et al., Pg. 46 ¶ 0323 - 0327, Pg. 47 ¶ 0329 - 0331, Pg. 49 ¶ 0341 - 0343) processing, by the computing system, (Lyman et al., Pg. 1 ¶ 0034 - Pg. 2 ¶ 0037, Pg. 4 ¶ 0050, Pg. 57 ¶ 0392, Pg. 57 ¶ 0397 - Pg. 58 ¶ 0398) the one or more images with an intent classification model to generate an intent classification, (Lyman et al., Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342 [“Each processing function 4008 can utilize a computer vision model trained to diagnose conditions of the corresponding type of image, for example, where each processing function 4008 is trained on a training set of images of the corresponding type as input data and on corresponding diagnosis data as output labels” and “multiple processing functions 4008 are performed in accordance with a hierarchy of processing functions 4008. For example, a first processing function 4008 trained to detect skin conditions is performed first on the captured image in response to determining the captured image is a picture of the skin. The diagnosis data generated by the first processing function can indicates a rash, and second processing functions 4008 trained to diagnose particular types of rashes can be selected to be performed next on the captured image to provide a more detailed diagnosis”]) wherein the intent classification indicates that the search query has a diagnostic search intent, (Lyman et al., Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342 [“Each processing function 4008 can utilize a computer vision model trained to diagnose conditions of the corresponding type of image, for example, where each processing function 4008 is trained on a training set of images of the corresponding type as input data and on corresponding diagnosis data as output labels” and “multiple processing functions 4008 are performed in accordance with a hierarchy of processing functions 4008. For example, a first processing function 4008 trained to detect skin conditions is performed first on the captured image in response to determining the captured image is a picture of the skin. The diagnosis data generated by the first processing function can indicates a rash, and second processing functions 4008 trained to diagnose particular types of rashes can be selected to be performed next on the captured image to provide a more detailed diagnosis”]) wherein the intent classification model was trained to determine a search intent of the user based on one or more features in an input image; (Lyman et al., Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342 [“Each processing function 4008 can utilize a computer vision model trained to diagnose conditions of the corresponding type of image, for example, where each processing function 4008 is trained on a training set of images of the corresponding type as input data and on corresponding diagnosis data as output labels”]) providing, by the computing system, (Lyman et al., Pg. 1 ¶ 0034 - Pg. 2 ¶ 0037, Pg. 4 ¶ 0050, Pg. 57 ¶ 0392, Pg. 57 ¶ 0397 - Pg. 58 ¶ 0398) the one or more images to a dermatology conditions classification model based on the intent classification; (Lyman et al., Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342 [“multiple processing functions 4008 are performed in accordance with a hierarchy of processing functions 4008. For example, a first processing function 4008 trained to detect skin conditions is performed first on the captured image in response to determining the captured image is a picture of the skin. The diagnosis data generated by the first processing function can indicates a rash, and second processing functions 4008 trained to diagnose particular types of rashes can be selected to be performed next on the captured image to provide a more detailed diagnosis”]) processing, by the computing system, (Lyman et al., Pg. 1 ¶ 0034 - Pg. 2 ¶ 0037, Pg. 4 ¶ 0050, Pg. 57 ¶ 0392, Pg. 57 ¶ 0397 - Pg. 58 ¶ 0398) the one or more images with the dermatology conditions classification model to generate one or more predicted condition classifications, (Lyman et al., Figs. 15A - 15C, Pg. 8 ¶ 0070 - 0075, Pg. 46 ¶ 0323 - 0326, Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342, Pg. 51 ¶ 0350 - 0353) wherein the one or more predicted condition classifications are descriptive of one or more candidate skin conditions; (Lyman et al., Figs. 15A - 15C, Pg. 8 ¶ 0070 - 0075, Pg. 46 ¶ 0323 - 0326, Pg. 47 ¶ 0328 - 0332, Pg. 49 ¶ 0341 - 0342, Pg. 51 ¶ 0350 - 0355) and providing, by the computing system, (Lyman et al., Pg. 1 ¶ 0034 - Pg. 2 ¶ 0037, Pg. 4 ¶ 0050, Pg. 57 ¶ 0392, Pg. 57 ¶ 0397 - Pg. 58 ¶ 0398) skin condition information associated with the one or more candidate skin conditions as an output. (Lyman et al., Figs. 15A - 15C, Pg. 8 ¶ 0070 - 0075, Pg. 46 ¶ 0323 - 0326, Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342, Pg. 51 ¶ 0350 - 0353) Peng et al. and Lyman et al. are combinable because they are both directed towards medical image processing systems that process dermatology images depicting skin to generate predicted diagnosis data for the dermatology images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Peng et al. with the teachings of Lyman et al. This modification would have been prompted in order to enhance the base device of Peng et al. with the well-known and applicable techniques Lyman et al. applied to a comparable device. Processing the one or more images with a skin classification model to determine the one or more images depict skin, processing the one or more images with an intent classification model to generate an intent classification and providing the one or more images to the dermatology conditions classification model based on the intent classification, as taught by Lyman et al., would enhance the base device of Peng et al. by reducing the total number of image processing operations it performs since only search query images determined to depict skin and have a diagnostic search intent would undergo processing by the dermatology conditions classification model thereby eliminating unnecessary image processing operations from being performed by the dermatology conditions classification model on search query images lacking depictions of skin and/or diagnostic search intent and improving the overall computational efficiency of the base device of Peng et al. This combination could be completed according to well-known techniques in the art and would likely yield predictable results, in that the one or more images would be processed with a skin classification model to determine the one or more images depict skin, processed with an intent classification model to generate an intent classification and provided to the dermatology conditions classification model based on the intent classification so as to help ensure that unnecessary image processing operations are not performed by the dermatology conditions classification model on search query images lacking depictions of skin and/or a diagnostic search intent and thereby improve the overall computational efficiency of the base device of Peng et al. Therefore, it would have been obvious to combine Peng et al. with Lyman et al. to obtain the invention as specified in claim 11.
- With regards to claim 12, Peng et al. in view of Lyman et al. disclose the method of claim 11, further comprising: obtaining, by the computing system, the skin condition information from a curated database, (Peng et al., Figs. 3, 8 & 9, Pg. 1 ¶ 0012 - 0013, Pg. 3 ¶ 0046 - 0047, Pg. 4 ¶ 0051, Pg. 6 ¶ 0062 - 0065) wherein the curated database comprises a plurality of condition datasets associated with a plurality of different skin conditions; (Peng et al., Figs. 3, 8 & 9, Pg. 1 ¶ 0012 - 0013, Pg. 3 ¶ 0040 - 0044 and 0046 - 0047, Pg. 4 ¶ 0049 - 0051, Pg. 6 ¶ 0060 and 0062 - 0065, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102) processing, by the computing system, the one or more images with a search engine to identify a plurality of different visual search results, (Peng et al., Abstract, Figs. 1 - 10 & 17, Pg. 1 ¶ 0003 - 0008 and 0013, Pg. 3 ¶ 0040 - 0044 and 0046, Pg. 4 ¶ 0050, Pg. 5 ¶ 0052 - 0055 and 0059, Pg. 6 ¶ 0061 - 0062, Pg. 8 ¶ 0084 - 0088 and 0094 - 0096) wherein the plurality of different visual search results are associated with a plurality of different web resources; (Peng et al., Figs. 3, 8 & 9, Pg. 1 ¶ 0013, Pg. 3 ¶ 0046 - 0047, Pg. 4 ¶ 0049 - 0051, Pg. 6 ¶ 0060 - 0063, Pg. 7 ¶ 0070, Pg. 8 ¶ 0094 - 0096 [“The reference library could be obtained from open sources, licensed from institutions such as universities or hospital systems, or a combination thereof… The reference library, in whole or in part, could be generated from open sources such as The Cancer Genome Atlas, California Tumor Tissue Registry, etc., or from particular medical institutions 320 and 322 such as hospitals or universities, the U.S. Government or department thereof, such as the Centers For Disease Control, National Institutes of Health, the U.S. Navy, or combinations thereof” and “The user is given a choice of databases to search in, by checking the box 910, 912 next the name of the institution or source of the images in the reference library”]) and providing, by the computing system, the plurality of different visual search results for display with the skin condition information. (Peng et al., Figs. 1, 2, 10 & 17, Pg. 1 ¶ 0003 and 0006 - 0012, Pg. 3 ¶ 0043 - 0044, Pg. 4 ¶ 0050, Pg. 6 ¶ 0063 - 0065, Pg. 7 ¶ 0070, Pg. 8 ¶ 0089 - 0092 and 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102)
- With regards to claim 13, Peng et al. in view of Lyman et al. disclose the method of claim 12, wherein the plurality of condition datasets were at least one of generated or reviewed by a licensed dermatologist. (Peng et al., Pg. 3 ¶ 0040 - 0043 and 0046 - 0047, Pg. 4 ¶ 0051, Pg. 5 ¶ 0054 and 0059, Pg. 6 ¶ 0063, Pg. 7 ¶ 0070, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102 [“generation of semantic similarity labels for medical images can be curated manually or generated from crowdsourced training based on a pathologist click behavior when viewing collections of medical images or other machine learning technique”, “reference library, in whole or in part, could be generated from open sources such as The Cancer Genome Atlas, California Tumor Tissue Registry, etc., or from particular medical institutions 320 and 322 such as hospitals or universities, the U.S. Government or department thereof, such as the Centers For Disease Control, National Institutes of Health, the U.S. Navy, or combinations thereof”, “Dr. Lee is a general dermatologist who has a patient with a suspicious mole. She has arrived at a diagnosis, but wants to double check her assessment… 2. A pivot table (e.g., FIG. 1) is presented of the e.g., 50 cases that were identified as morphologically similar. The table shows the diagnosis & percent of cases with that diagnosis. 3. If most of the identified cases are in line with her assessment, she's reassured that she made the right diagnosis. If most of historical cases fall into another diagnostic category, she can take another look at her current case and compare it to historical cases to make a final diagnosis… 4. She also wants to be able to browse the cases identified to compare them manually to the current case. Hence, she selects any of the cases in the pivot table and browses the images and reviews associated metadata.”])
- With regards to claim 14, Peng et al. in view of Lyman et al. disclose the method of claim 12, wherein providing, by the computing system, the plurality of different visual search results for display with the skin condition information comprises: ordering, by the computing system and via a ranking engine, the plurality of different visual search results based on: a determined visual similarity with the one or more images; (Peng et al., Abstract, Figs. 6, 7 & 10, Pg. 1 ¶ 0003 and 0007 - 0010, Pg. 5 ¶ 0054 - 0055 and 0059, Pg. 6 ¶ 0061 and 0063, Pg. 8 ¶ 0094 - 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102, Pg. 9 ¶ 0110) and a determined topic relevance based on whether the particular visual search result is associated with the one or more candidate skin conditions. (Peng et al., Abstract, Figs. 6, 7 & 10, Pg. 1 ¶ 0003 and 0008 - 0012, Pg. 5 ¶ 0054 - 0055 and 0059, Pg. 6 ¶ 0061 and 0063, Pg. 7 ¶ 0067 - 0070 and 0081, Pg. 8 ¶ 0094 - 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102, Pg. 9 ¶ 0110)
- With regards to claim 15, Peng et al. in view of Lyman et al. disclose the method of claim 11, wherein processing, by the computing system, the one or more images with the dermatology conditions classification model to generate the one or more predicted condition classifications comprises: processing, by the computing system, the one or more images with the dermatology conditions classification model to generate a plurality of predicted condition classifications, (Peng et al., Figs. 1, 2 & 10 - 12, Pg. 1 ¶ 0008 and 0012, Pg. 2 ¶ 0016 - 0020, Pg. 3 ¶ 0040 - 0044 and 0047, Pg. 4 ¶ 0050, Pg. 5 ¶ 0054, Pg. 6 ¶ 0062 - 0065, Pg. 7 ¶ 0069 - 0070, Pg. 8 ¶ 0088 - 0091 and 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0100) wherein each predicted condition classification is descriptive of a particular candidate skin condition; (Peng et al., Figs. 1, 2 & 10 - 12, Pg. 1 ¶ 0004, 0008 and 0012, Pg. 3 ¶ 0040 - 0044 and 0047, Pg. 4 ¶ 0050 - 0051, Pg. 6 ¶ 0062 - 0065, Pg. 8 ¶ 0088 - 0091 and 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102) obtaining, by the computing system, a plurality of skin information datasets associated with the plurality of predicted condition classifications, (Peng et al., Figs. 3, 8 & 9, Pg. 1 ¶ 0008 and 0012 - 0013, Pg. 3 ¶ 0040 - 0044 and 0046 - 0047, Pg. 4 ¶ 0049 - 0051, Pg. 6 ¶ 0060 and 0062 - 0065, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102) wherein each skin information dataset of the plurality of skin information datasets is associated with a different candidate skin condition; (Peng et al., Figs. 1, 3 - 5 & 10, Pg. 1 ¶ 0008 and 0011 - 0013, Pg. 3 ¶ 0040 - 0044 and 0046 - 0047, Pg. 4 ¶ 0051 - Pg. 5 ¶ 0055, Pg. 5 ¶ 0059 - Pg. 6 ¶ 0060, Pg. 6 ¶ 0062 - Pg. 7 ¶ 0070, Pg. 8 ¶ 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102) and providing, by the computing system, the plurality of skin information datasets for display. (Peng et al., Figs. 1, 2, 10 & 17, Pg. 1 ¶ 0003 and 0006 - 0012, Pg. 3 ¶ 0043 - 0044, Pg. 4 ¶ 0050, Pg. 6 ¶ 0063 - 0065, Pg. 7 ¶ 0070, Pg. 8 ¶ 0089 - 0092 and 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102) Peng et al. fail to disclose explicitly displaying via a carousel interface. However, the Examiner takes official notice of the fact that displaying search results, such as datasets or images, via a carousel interface is notoriously well-known in the art. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combined teachings of Peng et al. in view of Lyman et al. to include a carousel interface for displaying the obtained datasets. This modification would have been prompted in order to substitute the retrieval panes or pivot tables of Peng et al. for a notoriously well-known carousel interface. A notoriously well-known carousel interface could be substituted in place of the retrieval panes or pivot tables of Peng et al. using well-known techniques in the art and would likely yield predictable results, in that, in the combination, the plurality of skin information datasets would be displayed via a carousel interface. This combination could be completed according to well-known techniques in the art and would likely yield predictable results, in that the combined base device would provide the plurality of skin information datasets for display via a carousel interface. Therefore, it would have been obvious to combine Peng et al. in view of Lyman et al. with the notoriously well-known technique of displaying datasets via a carousel interface to obtain the invention as specified in claim 15.
- With regards to claim 16, Peng et al. in view of Lyman et al. disclose the method of claim 11, wherein obtaining, by the computing system, the search query comprises: obtaining, by the computing system, the search query via a user interface of a visual search application; (Peng et al., Abstract, Figs. 1, 2, 6 - 10 & 17, Pg. 1 ¶ 0003 and 0006, Pg. 3 ¶ 0040 - 0043, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0061 - 0063, Pg. 7 ¶ 0082 - 0083, Pg. 8 ¶ 0092 - 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0104, Pg. 9 ¶ 0109 - 0110) and wherein providing, by the computing system, the skin condition information associated with the one or more candidate skin conditions comprises: providing, by the computing system, the skin condition information for display via the user interface of the visual search application. (Peng et al., Figs. 1, 2, 10 & 17, Pg. 1 ¶ 0003 and 0006 - 0012, Pg. 3 ¶ 0043 - 0044, Pg. 4 ¶ 0050, Pg. 6 ¶ 0063 - 0065, Pg. 7 ¶ 0070, Pg. 8 ¶ 0089 - 0092 and 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102) In addition, analogous art Lyman et al. disclose wherein obtaining, by the computing system, the search query comprises: obtaining, by the computing system, the search query via a user interface of a visual search application; (Lyman et al., Figs. 15A - 15C, Pg. 2 ¶ 0037, Pg. 4 ¶ 0050, Pg. 5 ¶ 0054 - 0055, Pg. 46 ¶ 0323 - 0324, Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0339 - 0342, Pg. 57 ¶ 0392 - Pg. 58 ¶ 0398) and wherein providing, by the computing system, the skin condition information associated with the one or more candidate skin conditions comprises: providing, by the computing system, the skin condition information for display via the user interface of the visual search application. (Lyman et al., Figs. 15A - 16A & 16D, Pg. 2 ¶ 0037, Pg. 4 ¶ 0050, Pg. 5 ¶ 0054 - 0055, Pg. 8 ¶ 0070 - 0075, Pg. 46 ¶ 0323 - 0326, Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342, Pg. 51 ¶ 0350 - 0353, Pg. 57 ¶ 0392 - Pg. 58 ¶ 0398)
- With regards to claim 17, Peng et al. disclose instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, (Peng et al., Abstract, Fig. 8, Pg. 1 ¶ 0003, Pg. 2 ¶ 0016, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0061 and 0065, Pg. 8 ¶ 0088, Pg. 9 ¶ 0105 [“computer system”]) the operations comprising: obtaining a search query (Peng et al., Abstract, Figs. 1, 2, 9 - 12 & 17, Pg. 1 ¶ 0003 - 0004, Pg. 3 ¶ 0040 - 0042, Pg. 4 ¶ 0049 - 0051, Pg. 6 ¶ 0062 - 0065, Pg. 7 ¶ 0069 - 0071, Pg. 8 ¶ 0092 - 0099, Pg. 9 ¶ 0102 - 0105) from a user computing system, (Peng et al., Abstract, Fig. 8, Pg. 1 ¶ 0003, Pg. 2 ¶ 0016, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0061 and 0065, Pg. 8 ¶ 0088, Pg. 9 ¶ 0105 [“computer system”]) wherein the search query comprises one or more images, wherein the one or more images depict one or more body parts of a user; (Peng et al., Abstract, Figs. 1, 2, 9 - 12 & 17, Pg. 1 ¶ 0003 - 0004, Pg. 3 ¶ 0040 - 0042, Pg. 4 ¶ 0049 - 0051, Pg. 6 ¶ 0062 - 0065, Pg. 7 ¶ 0069 - 0071, Pg. 8 ¶ 0092 - 0099, Pg. 9 ¶ 0102 - 0105) wherein the search query has a diagnostic search intent; (Peng et al., Figs. 1, 2, 10, 11, 13 & 14, Pg. 1 ¶ 0008 and 0011 - 0012, Pg. 2 ¶ 0017 - 0020, Pg. 3 ¶ 0040 - 0044 and 0047, Pg. 4 ¶ 0050 - 0051, Pg. 5 ¶ 0054, Pg. 6 ¶ 0062 - Pg. 7 ¶ 0068, Pg. 7 ¶ 0070, Pg. 8 ¶ 0089 - 0091, 0093 and 0096 - 0099, Pg. 9 ¶ 0102 and 0110) providing the one or more images to a medical condition classification model based on the diagnostic search intent; (Peng et al., Abstract, Figs. 6 - 11, Pg. 1 ¶ 0003 - 0005, Pg. 2 ¶ 0016 - 0020, Pg. 3 ¶ 0045 - 0047, Pg. 4 ¶ 0049 - 0050, Pg. 5 ¶ 0054 - 0059, Pg. 6 ¶ 0061 - 0063 and 0065, Pg. 7 ¶ 0083 - Pg. 8 ¶ 0086, Pg. 8 ¶ 0088) processing the one or more images with the medical conditions classification model to generate one or more predicted condition classifications, (Peng et al., Figs. 1, 2 & 10 - 12, Pg. 1 ¶ 0008 and 0012, Pg. 2 ¶ 0016 - 0020, Pg. 3 ¶ 0040 - 0044 and 0047, Pg. 4 ¶ 0050, Pg. 5 ¶ 0054, Pg. 6 ¶ 0062 - 0065, Pg. 7 ¶ 0069 - 0070, Pg. 8 ¶ 0088 - 0091 and 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0100) wherein the one or more predicted condition classifications are descriptive of one or more candidate medical conditions determined to be potentially depicted in the one or more images; (Peng et al., Figs. 1, 2 & 10 - 12, Pg. 1 ¶ 0008 and 0012, Pg. 3 ¶ 0040 - 0044 and 0047, Pg. 4 ¶ 0050, Pg. 6 ¶ 0063 - 0065, Pg. 7 ¶ 0070, Pg. 8 ¶ 0088 - 0091 and 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0100) obtaining condition information for the one or more candidate medical conditions, (Peng et al., Figs. 3, 8 & 9, Pg. 1 ¶ 0012 - 0013, Pg. 3 ¶ 0046 - 0047, Pg. 4 ¶ 0051, Pg. 6 ¶ 0062 - 0065) wherein the condition information comprises one or more example images of the particular candidate medical condition and a condition name; (Peng et al., Figs. 1, 2 & 10 - 12, Pg. 1 ¶ 0006 - 0008 and 0011 - 0013, Pg. 3 ¶ 0040 - 0044 and 0046 - 0047, Pg. 4 ¶ 0050 - 0051, Pg. 6 ¶ 0062 - 0065, Pg. 7 ¶ 0069 - 0070, Pg. 8 ¶ 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102) processing the one or more images with a search engine to determine one or more visual search results, (Peng et al., Abstract, Figs. 1 - 10 & 17, Pg. 1 ¶ 0003 - 0008 and 0013, Pg. 3 ¶ 0040 - 0044 and 0046, Pg. 4 ¶ 0050, Pg. 5 ¶ 0052 - 0055 and 0059, Pg. 6 ¶ 0061 - 0062, Pg. 8 ¶ 0084 - 0088 and 0094 - 0096) wherein the one or more visual search results are determined based on a visual feature similarity with the one or more images; (Peng et al., Abstract, Figs. 6, 7 & 10, Pg. 1 ¶ 0003 and 0007 - 0010, Pg. 5 ¶ 0054 - 0055 and 0059, Pg. 6 ¶ 0061 and 0063, Pg. 8 ¶ 0094 - 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102, Pg. 9 ¶ 0110) and providing the one or more visual search results and the condition information to the user computing system. (Peng et al., Figs. 1, 2, 10 & 17, Pg. 1 ¶ 0003 and 0006 - 0012, Pg. 3 ¶ 0043 - 0044, Pg. 4 ¶ 0050, Pg. 6 ¶ 0063 - 0065, Pg. 7 ¶ 0070, Pg. 8 ¶ 0089 - 0092 and 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0102) Peng et al. fail to disclose explicitly one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations; processing the one or more images with an intent classification model to generate an intent classification, wherein the intent classification indicates that the search query has a diagnostic search intent, wherein the intent classification model was trained to determine a search intent of the user based on one or more features in an input image; and providing the images to the medical condition classification model based on the intent classification. Pertaining to analogous art, Lyman et al. disclose one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, (Lyman et al., Pg. 1 ¶ 0034 - Pg. 2 ¶ 0037, Pg. 4 ¶ 0050, Pg. 57 ¶ 0392, Pg. 57 ¶ 0397 - Pg. 58 ¶ 0398) the operations comprising: obtaining a search query from a user computing system, wherein the search query comprises one or more images, wherein the one or more images depict one or more body parts of a user; (Lyman et al., Figs. 1 - 3 & 15A - 16A, Pg. 14 ¶ 0108 - 0110, Pg. 46 ¶ 0323 - 0325, Pg. 46 ¶ 0327 - Pg. 47 ¶ 0332, Pg. 48 ¶ 0334 - 0336, Pg. 49 ¶ 0340 - 0342) processing the one or more images with an intent classification model to generate an intent classification, (Lyman et al., Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342 [“Each processing function 4008 can utilize a computer vision model trained to diagnose conditions of the corresponding type of image, for example, where each processing function 4008 is trained on a training set of images of the corresponding type as input data and on corresponding diagnosis data as output labels” and “multiple processing functions 4008 are performed in accordance with a hierarchy of processing functions 4008. For example, a first processing function 4008 trained to detect skin conditions is performed first on the captured image in response to determining the captured image is a picture of the skin. The diagnosis data generated by the first processing function can indicates a rash, and second processing functions 4008 trained to diagnose particular types of rashes can be selected to be performed next on the captured image to provide a more detailed diagnosis”]) wherein the intent classification indicates that the search query has a diagnostic search intent, (Lyman et al., Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342 [“Each processing function 4008 can utilize a computer vision model trained to diagnose conditions of the corresponding type of image, for example, where each processing function 4008 is trained on a training set of images of the corresponding type as input data and on corresponding diagnosis data as output labels” and “multiple processing functions 4008 are performed in accordance with a hierarchy of processing functions 4008. For example, a first processing function 4008 trained to detect skin conditions is performed first on the captured image in response to determining the captured image is a picture of the skin. The diagnosis data generated by the first processing function can indicates a rash, and second processing functions 4008 trained to diagnose particular types of rashes can be selected to be performed next on the captured image to provide a more detailed diagnosis”]) wherein the intent classification model was trained to determine a search intent of the user based on one or more features in an input image; (Lyman et al., Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342 [“Each processing function 4008 can utilize a computer vision model trained to diagnose conditions of the corresponding type of image, for example, where each processing function 4008 is trained on a training set of images of the corresponding type as input data and on corresponding diagnosis data as output labels”]) providing the one or more images to a medical condition classification model based on the intent classification; (Lyman et al., Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342 [“multiple processing functions 4008 are performed in accordance with a hierarchy of processing functions 4008. For example, a first processing function 4008 trained to detect skin conditions is performed first on the captured image in response to determining the captured image is a picture of the skin. The diagnosis data generated by the first processing function can indicates a rash, and second processing functions 4008 trained to diagnose particular types of rashes can be selected to be performed next on the captured image to provide a more detailed diagnosis”]) and processing the one or more images with the medical conditions classification model to generate one or more predicted condition classifications, (Lyman et al., Figs. 15A - 15C, Pg. 8 ¶ 0070 - 0075, Pg. 46 ¶ 0323 - 0326, Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342, Pg. 51 ¶ 0350 - 0353) wherein the one or more predicted condition classifications are descriptive of one or more candidate medical conditions determined to be potentially depicted in the one or more images. (Lyman et al., Figs. 15A - 15C, Pg. 8 ¶ 0070 - 0075, Pg. 46 ¶ 0323 - 0326, Pg. 47 ¶ 0328 - 0331, Pg. 49 ¶ 0341 - 0342, Pg. 51 ¶ 0350 - 0353) Peng et al. and Lyman et al. are combinable because they are both directed towards medical image processing systems that process dermatology images depicting skin to generate predicted diagnosis data for the dermatology images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Peng et al. with the teachings of Lyman et al. A first modification would have been prompted in order to enhance the base device of Peng et al. with the well-known and applicable technique Lyman et al. applied to a comparable device. Utilizing one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, as taught by Lyman et al., to implement operations of the base device of Peng et al. would enhance the base device of Peng et al. by ensuring that its operations are carried out accurately and efficiently at high-computational speed on computer architecture, by facilitating widespread distribution of the base device of Peng et al. to millions of potential end-users with access to a computer and by simplifying the process of making revisions, modifications and/or updates to the operations of the base device of Peng et al. A second modification would have been prompted in order to enhance the base device of Peng et al. with the well-known and applicable technique Lyman et al. applied to a comparable device. Processing the one or more images with an intent classification model to generate an intent classification and providing the one or more images to the medical conditions classification model based on the intent classification, as taught by Lyman et al., would enhance the base device of Peng et al. by reducing the total number of image processing operations it performs since only search query images determined to have a diagnostic search intent would undergo processing by the medical conditions classification model thereby eliminating unnecessary image processing operations from being performed by the medical conditions classification model on search query images lacking diagnostic search intent and improving the overall computational efficiency of the base device of Peng et al. This combination could be completed according to well-known techniques in the art and would likely yield predictable results, in that one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations would be utilized to implement the base device of Peng et al. so as to ensure that its operations are carried out accurately and efficiently at high-computational speed on computer architecture, to facilitate widespread distribution of the base device of Peng et al. to millions of potential end-users with access to a computer and to simplify the process of making revisions, modifications and/or updates to the operations of the base device of Peng et al. and in that the one or more images would be processed with an intent classification model to generate an intent classification and provided to the medical conditions classification model based on the intent classification so as to help ensure that unnecessary image processing operations are not performed by the medical conditions classification model on search query images lacking a diagnostic search intent and thereby improve the overall computational efficiency of the base device of Peng et al. Therefore, it would have been obvious to combine Peng et al. with Lyman et al. to obtain the invention as specified in claim 17.
- With regards to claim 18, Peng et al. in view of Lyman et al. disclose the one or more non-transitory computer-readable media of claim 17, wherein processing the one or more images with a search engine to determine one or more visual search results comprises: determining a plurality of candidate visual search results based on the one or more images; (Peng et al., Abstract, Figs. 1 - 10 & 17, Pg. 1 ¶ 0003 - 0008 and 0013, Pg. 3 ¶ 0040 - 0044 and 0046, Pg. 4 ¶ 0050, Pg. 5 ¶ 0052 - 0055 and 0059, Pg. 6 ¶ 0061 - 0062, Pg. 8 ¶ 0084 - 0088 and 0094 - 0096) determining one or more anatomy visual search results of the plurality of candidate visual search results, (Peng et al., Figs. 3 & 13 - 15, Pg. 3 ¶ 0046 - 0047, Pg. 4 ¶ 0050 - 0051, Pg. 5 ¶ 0059 - Pg. 6 ¶ 0060, Pg. 6 ¶ 0063 - 0065, Pg. 7 ¶ 0067 - 0070, Pg. 8 ¶ 0093 - 0096 and 0099, Pg. 9 ¶ 0102 and 0110) wherein the one or more anatomy visual search results comprise one or more anatomy images that depict a human boy part; (Peng et al., Figs. 3 & 13 - 15, Pg. 3 ¶ 0046 - 0047, Pg. 4 ¶ 0050 - 0051, Pg. 5 ¶ 0059 - Pg. 6 ¶ 0060, Pg. 6 ¶ 0063 - 0065, Pg. 7 ¶ 0067 - 0070, Pg. 8 ¶ 0093 - 0096 and 0099, Pg. 9 ¶ 0102 and 0110) and wherein the one or more visual search results comprise the one or more anatomy visual search results. (Peng et al., Figs. 1 - 3, 10 - 12 & 17, Pg. 1 ¶ 0006 - 0012, Pg. 3 ¶ 0043 - 0047, Pg. 4 ¶ 0049 - 0051, Pg. 5 ¶ 0054, Pg. 6 ¶ 0061 - 0065, Pg. 7 ¶ 0067 - 0070, Pg. 8 ¶ 0094 - 0096, Pg. 9 ¶ 0101 - 0104 and 0110)
- With regards to claim 19, Peng et al. in view of Lyman et al. disclose the one or more non-transitory computer-readable media of claim 17, wherein processing the one or more images with a search engine to determine one or more visual search results comprises: determining a plurality of candidate visual search results based on the one or more images; (Peng et al., Abstract, Figs. 1 - 10 & 17, Pg. 1 ¶ 0003 - 0008 and 0013, Pg. 3 ¶ 0040 - 0044 and 0046, Pg. 4 ¶ 0050, Pg. 5 ¶ 0052 - 0055 and 0059, Pg. 6 ¶ 0061 - 0062, Pg. 8 ¶ 0084 - 0088 and 0094 - 0096) determining one or more particular candidate search results of the plurality of candidate visual search results are associated with the one or more candidate medical conditions; (Peng et al., Abstract, Figs. 1, 2, 6, 7, 10 & 11, Pg. 1 ¶ 0003 - 0012, Pg. 3 ¶ 0043 - 0044, Pg. 4 ¶ 0050, Pg. 5 ¶ 0052 - 0055 and 0059, Pg. 6 ¶ 0061 - 0065, Pg. 7 ¶ 0067 - 0070, Pg. 8 ¶ 0084 - 0087 and 0094 - 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0104) and adjusting the ranking of the plurality of candidate visual search results based on determining the one or more particular candidate search results are associated with the one or more candidate medical conditions. (Peng et al., Abstract, Figs. 6, 7 & 10, Pg. 1 ¶ 0003 and 0005 - 0012, Pg. 5 ¶ 0054 - 0055, Pg. 5 ¶ 0059 - Pg. 6 ¶ 0061, Pg. 6 ¶ 0063, Pg. 7 ¶ 0068 - 0070 and 0081, Pg. 8 ¶ 0096, Pg. 9 ¶ 0100 - 0104 [“The computer system is trained to find one or more similar medical images in the reference library system which are similar to the input image. The reference library is represented as an embedding of each of the medical images projected in a feature space having a plurality of axes, wherein the embedding is characterized by two aspects of a similarity ranking: (1) visual similarity, and (2) semantic similarity such that neighboring images in the feature space are visually similar and semantic information is represented by the axes of the feature space”])
- With regards to claim 20, Peng et al. in view of Lyman et al. disclose the one or more non-transitory computer-readable media of claim 17, wherein the one or more visual search results are provided in a first panel of a search results interface, (Peng et al., Figs. 1, 2, 10, 11 & 17, Pg. 1 ¶ 0006 - 0008 and 0011, Pg. 2 ¶ 0022 - 0024 and 0032 - 0033, Pg. 3 ¶ 0043 - 0044, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0061 - 0065, Pg. 8 ¶ 0086 - 0087 and 0094 - 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0100, Pg. 9 ¶ 0102 and 0110) and wherein the condition information is provided in a second panel of the search results interface. (Peng et al., Figs. 1, 2, 10 & 11, Pg. 3 ¶ 0043 - 0044, Pg. 4 ¶ 0049 - 0050, Pg. 6 ¶ 0063 - 0065, Pg. 8 ¶ 0094 - 0096, Pg. 8 ¶ 0099 - Pg. 9 ¶ 0100)
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Peng et al. U.S. Publication No. 2021/0019342 A1 in view of Lyman et al. U.S. Publication No. 2020/0160980 A1 as applied to claim 9 above, and further in view of Kondo et al. U.S. Publication No. 2017/0091413 A1.
- With regards to claim 10, Peng et al. in view of Lyman et al. disclose the system of claim 9, wherein the operations further comprise: obtaining a selection input associated with the selectable user interface element, wherein the selection input is descriptive of a selection of the selectable user interface element; (Peng et al., Figs. 1, 10 & 17, Pg. 1 ¶ 0008 - 0012, Pg. 4 ¶ 0048 - 0049, Pg. 6 ¶ 0061 - 0064, Pg. 8 ¶ 0086 - 0087, 0094 - 0096 and 0099) and processing the particular medical condition with a search engine to determine a plurality of updated search results associated with the particular medical condition; (Peng et al., Figs. 1, 2, 10 & 11, Pg. 1 ¶ 0009 - 0012, Pg. 4 ¶ 0048, Pg. 6 ¶ 0063 - 0064, Pg. 8 ¶ 0086 - 0087, 0094 - 0096 and 0099, Pg. 9 ¶ 0110 [“task bar 1004 provides tools for filtering the retrieved images, e.g., on the basis of metadata associated with each of the images. For example, the user could select filters such that only images of cancerous lesions are displayed, only images from male or female patients, only images with a survival time of less than 5 years, etc. In the results region 1010, the user is given an option to show statistics of the returned images, such as the top diagnoses for all 50 images returned, which are listed in the region 1010. A drop-down box 1012 allow for the user to select other statistics or summaries of the returned images. The menu of options available in the drop-down box will vary on the type of images and persons skilled in the art will be able to arrive at a suitable menu of options depending on implementation details which are not particularly important. Examples of such options include Diagnosis” and “tools on the display could interactively adjust the similarity metric based on the user's input, to facilitate an iterative search. In particular, the result set can be refined based on the previous choices of the user. If, for example, the user selected an image of similar cancer grade (but dissimilar histological features), more images of the similar cancer grade could be shown by emphasizing that particular dimension in the metric embedding space for the neighbor list search”]) and providing the plurality of updated search results for display. (Peng et al., Figs. 1, 2, 10, 11 & 17, Pg. 1 ¶ 0009 - 00012, Pg. 2 ¶ 0022, Pg. 4 ¶ 0048 - 0050, Pg. 6 ¶ 0061 - 0064, Pg. 8 ¶ 0086 - 0087, 0094 - 0096 and 0099, Pg. 9 ¶ 0110) Peng et al. fail to disclose explicitly processing a condition name of the particular medical condition to determine the plurality of updated search results. Pertaining to analogous art, Kondo et al. disclose processing a condition name of the particular medical condition with a search engine to determine a plurality of updated search results associated with the particular medical condition; (Kondo et al., Figs. 45 & 46, Pg. 2 ¶ 0053 - 0054, Pg. 5 ¶ 0125 - 0128, Pg. 10 ¶ 0196 - Pg. 11 ¶ 0203, Pg. 18 ¶ 0300 - Pg. 19 ¶ 0308, Pg. 20 ¶ 0326 - Pg. 21 ¶ 0329, Pg. 21 ¶ 0331, Pg. 24 ¶ 0381 - Pg. 25 ¶ 0389) and providing the plurality of updated search results for display. (Kondo et al., Abstract, Figs. 45 & 46, Pg. 2 ¶ 0053 - 0054, Pg. 5 ¶ 0128, Pg. 10 ¶ 0196 - Pg. 11 ¶ 0205, Pg. 19 ¶ 0307 - 0309, Pg. 20 ¶ 0327 - Pg. 21 ¶ 0331, Pg. 24 ¶ 0381 - Pg. 25 ¶ 0389) Peng et al. in view of Lyman et al. and Kondo et al. are combinable because they are all directed towards medical image processing systems that process search query images to generate predicted diagnosis data for the search query images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the combined teachings of Peng et al. in view of Lyman et al. with the teachings of Kondo et al. This modification would have been prompted in order to substitute the search results filtering/refining technique of Peng et al. for the search results refining/updating process of Kondo et al. The search results refining/updating process of Kondo et al. could be substituted in place of the search results filtering/refining technique of Peng et al. using well-known techniques in the art and would likely yield predictable results, in that, in the combination, the search results refining/updating process of Kondo et al., which processes a condition name with a search engine, would be utilized to determine the plurality of updated search results associated with the particular medical condition. Furthermore, this modification would have been prompted by the teachings and suggestions of Peng et al. that their user interface may provide tools for filtering and/or refining searches and that the tools may filter retrieved images on the basis of metadata associated with the retrieved images so that, for example, only images of a certain diagnosis are displayed, see at least page 2 paragraph 0022, page 4 paragraphs 0048 - 0049, page 6 paragraph 0063 and page 8 paragraphs 0087 and 0094 - 0096 of Peng et al. This combination could be completed according to well-known techniques in the art and would likely yield predictable results, in that the combined base device would process a condition name of the particular medical condition with a search engine to determine the plurality of updated search results. Therefore, it would have been obvious to combine Peng et al. in view of Lyman et al. with Kondo et al. to obtain the invention as specified in claim 10.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Chen et al. U.S. Publication No. 2010/0302358 A1; which is directed towards an automated skin lesion assessment system, wherein a diagnosis for a user’s skin condition is determined based on comparing an image of the user’s skin condition against images of a plurality of diagnosed skin lesions.
Dunn et al. International Publication No. WO 2021/050928 A1; which is directed towards a system and method for diagnosing skin diseases, wherein a machine-learned skin condition classification model processes images depicting a portion of a patient’s skin to determine skin condition classifications for the portion of the patient’s skin.
Kannan et al. U.S. Publication No. 2019/0311814 A1; which is directed towards systems and methods for responding to a healthcare inquiry, wherein an intent of a user’s healthcare inquiry is classified, if the intent is dermatological in nature the user is requested to upload an image of their skin area to be diagnosed and, once the image is uploaded, a machine-learned image classifier produces a differential diagnosis of dermatological disease based on the image.
Rahman et al. U.S. Publication No. 2021/0118550 A1; which is directed towards a system and method for diagnosing a skin lesion, wherein a dermoscopic query image is compared to a plurality of images of pathologically confirmed types of skin lesions to determine whether the dermoscopic query image correlates to a skin cancer type.
Yang et al. U.S. Publication No. 2013/0013578 A1; which is directed towards a system and method for image retrieval, wherein a query image is provided to a search engine, the search engine determines a search intent from the query image and the search engine performs a search of an image database based on the search intent.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERIC RUSH whose telephone number is (571) 270-3017. The examiner can normally be reached 9am - 5pm Monday - Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270 - 5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ERIC RUSH/Primary Examiner, Art Unit 2677
Prosecution Insights