Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Claims 1-20 are pending. Claims 1, 11, and 16 are independent.
This Application was published as US 20240304176.
Apparent priority is 8 March 2023.
Response to Arguments
35 USC 103
Applicant’s arguments with respect to 35 USC 103 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 7, 8, 11, and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang et al. ("SVGAN: Semi-supervised Generative Adversarial Network for Image Captioning") in view of Selivanov et al. (“Medical Image Captioning via Generative Pretrained Transformers”) and Li et al. (US 20160364529 A1).
Regarding claim 1, Zhang discloses: 1. A method comprising: obtaining, via communications hardware of a voice intelligence manager, (Abstract mentions a computer, and the structure in Fig. 1 is clearly performed on computer hardware.)
a piece of data associated with a key performance indicator; ("Figure 2. Encoding area of image diagram." - Fig. 2 shows that an image (piece of data) is the input.)
identifying, by a feature identification engine of the voice intelligence manager, an actual feature in the piece of data using a generative adversarial network (GAN), wherein the actual feature represents a data point indicative of the key performance indicator (KPI); (Figure 1 shows a GAN. Fig. 1 shows "person" is identified, which is an actual feature in the data. )
identifying, by the feature identification engine, a trend feature indicating a summary of a pattern that is relative to a KPI threshold, wherein the trend feature comprises a plurality of data points overlapping at least in part with the data point; (not explicitly disclosed)
generating, by a summary generation engine of the voice intelligence manager and based on the actual feature, a first summary of the piece of data, wherein the first summary indicates the data trend feature associated with the key performance indicator; (Fig. 7 shows that the output is a caption sentence. Pg. 2, section B describes that the attributes (which would include “person”) are input to the graph embedding which is used to determine the caption.)
and causing, by the summary generation engine, display of the first summary. (Fig. 8 shows a summary displayed. )
Under the broadest reasonable interpretation, a key performance indicator could be any indicator. For example, fig. 8 shows captions which include colors. The color could be considered a key performance indicator. However, Zhang does not disclose a key performance indicator related to a patient’s health, as is discussed in [0004] of the instant application specification. Although this is a non-limiting example, for the purposes of compact prosecution, Zhang is not relied upon as teaching a key performance indicator. Zhang also does not disclose identifying, by the feature identification engine, a trend feature indicating a summary of a pattern that is relative to a KPI threshold, wherein the trend feature comprises a plurality of data points overlapping at least in part with the data point.
Selivanov discloses: obtaining, via communications hardware of a voice intelligence manager, a piece of data associated with a key performance indicator; (“Show Attend and Tell (SAT)11 is an attention-based image caption generation neural net. Attention-based technique allows to get well interpretable results, which can be utilized by radiologist to ensure their findings on X-Ray.” Pg. 3, section 2.1 – See also Table 2, which shows Chest X-Ray as the piece of data. Any of the detected conditions read on a key performance indicator.)
wherein the actual feature represents a data point indicative of the key performance indicator (Table 2 shows several descriptions of the x-rays which are features representing a data point indicative of the key performance indicator. For example, “pleural effusion present. lung opacity present” in the second example.)
wherein the first summary indicates the data point or a pattern associated with the key performance indicator. (Table 2 shows in the examples that a summary caption is generated which includes the feature description.)
Zhang and Selivanov are considered analogous art to the claimed invention because they disclose image captioning systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system of Zhang to use medical images as the input in order to detect a diagnosis as the caption output, as taught by Selivanov. Doing so would have been beneficial in order to ease clinical workflows and improve care quality and standardization. (“We propose to apply a model that works perfectly on non-medical data, to the medical data.” Selivanov, Pg. 1, section 1; “Providing automated support for this task has the potential to ease clinical workflows and improve both care quality and standardization.” Selivanov, Pg. 1, section 1.)
Selivanov does not disclose identifying, by the feature identification engine, a trend feature indicating a summary of a pattern that is relative to a KPI threshold, wherein the trend feature comprises a plurality of data points overlapping at least in part with the data point;
Li discloses: identifying, by the feature identification engine, a trend feature indicating a summary of a pattern that is relative to a KPI threshold, wherein the trend feature comprises a plurality of data points overlapping at least in part with the data point; ([0018] According to a second aspect, an information exchanging method is provided, where the method includes: acquiring increments of multiple medical images of a same region of a first object relative to a reference image, where the multiple medical images are stored according to the method in the first aspect or any possible implementation manner of the first to sixth possible implementation manners; determining a change trend of the multiple medical images according to the increments of the multiple medical images relative to the reference image; when the change trend of the multiple medical images exceeds a preset threshold, sending first information to a terminal of the first object, where the first information is used to indicate the change trend of the multiple medical images; and sending appointment information of a medical institution to the terminal of the first object, where the appointment information is used to indicate an available medical resource of the medical institution.)
wherein the first summary indicates the data trend feature associated with the key performance indicator (“[0125] The first information may be the change trend of the multiple medical images, or may be information that the change trend of the multiple medical images exceeds the preset threshold. In addition, the first information may be output in a form of an examination report, and the first information may also be displayed on a screen.”)
Zhang, Selivanov, and Li are considered analogous art to the claimed invention because they disclose image analysis systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination to use multiple images to determine a change over time and detect when a trend exceeds a threshold, as taught by Li. Doing so would have been beneficial in order to automatically schedule medical appointments when needed. (Li [0018].)
Regarding claim 7, Zhang discloses: 7. The method of claim 1, wherein: the GAN comprises a generator and a discriminator, (Fig. 1 shows a generator and discriminator. )
and identifying the actual feature of the piece of data using the GAN further comprises: by the generator: parsing the piece of data to identify a potential feature, and classifying the potential feature with a description; ("In the actual experiment process, the generator uses the LSTM network[10]. At each time step, the LSTM outputs the probability distribution of all words in the vocabulary according to the image characteristics and the previously generated words. " Pg. 2, Section B. – each word is a potential feature, and the probability describes the feature)
and by the discriminator: analyzing the piece of data, the potential feature, and the description of the potential feature, and approving or rejecting based on analyzing the piece of data, the potential feature, and the description of the potential feature, the potential feature as the actual feature of the piece of data. ("The discriminator is also composed of LSTM[11-12], which tries to distinguish the sentence from the corpus or from the generator. " Pg. 3, Section B. The sentence contains both the potential feature and the description of the potential feature; for example Fig. 8 shows “A blue keyboard”. Keyboard is a potential feature and blue is a description. Fig. 5 shows that the data is paired with the description.)
Regarding claim 8, Zhang discloses: 8. The method of claim 7, further comprising: receiving, by the communications hardware, a feedback for the first summary; ("Use ImageNet to pre-train a visual detector to detect the visual concepts in each picture. If the generated sentences contain visual concepts, a reward will be given." Pg. 3, Section C. - the reward is a feedback that is received.)
training, by the feature identification engine, the GAN using the feedback to obtain an updated GAN; and identifying, by the feature identification engine, actual features of subsequently received pieces of data using the updated GAN. ("Use the corpus to train a conditional generative adversarial network, the condition is the image feature, the generator and the discriminator are both LSTM." Pg. 3, Section C. - Fig. 7 further shows the training process including the reward.; Pg. 2, Section III. further describes the training set which would include subsequent pieces of data.)
Claim 11 is an apparatus claim with limitations corresponding to the limitations of Claim 1 and is rejected under similar rationale.
Claim 16 is a computer program product claim with limitations corresponding to the limitations of Claim 1 and is rejected under similar rationale. Additionally, “at least one non-transitory computer-readable storage medium” of the Claim are taught by Zhang. (“In order to improve the accuracy of generated sentences, we introduced a memory unit to store the information after modeling.” Pg. 2 Section B)
Claim(s) 2-6, 12-15, and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Selivanov and Li as applied in claim 1 above, in further view of Gao (US 20230103340 A1).
Regarding claim 2, Zhang does not disclose the additional limitations. Neither does Selivanov or Li.
Gao discloses: 2. The method of claim 1, further comprising: generating, by the summary generation engine, a summary audio file using the first summary; and causing, by the summary generation engine, playback of the summary audio file. ("[0155] In a possible implementation, since the generated image caption information is text information, in order to facilitate the target object to receive the image caption information, the computer device can convert text type image caption information into voice type image caption information based on the text-to-speech (TTS) technology, and transmit the image caption information to the target object in a form of voice playback." )
Zhang, Selivanov, Li, and Gao are considered analogous art to the claimed invention because they disclose image analysis systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination with TTS technology as taught by Gao. Doing so would have been beneficial so that visually impaired people could receive the data. (Gao [0060])
Regarding claim 3, Zhang discloses: 3. The method of claim 2, wherein the piece of data comprises at least one of an image, an audio recording, or a video. ("Figure 2. Encoding area of image diagram." - Fig. 2 shows that an image (piece of data) is the input. )
Regarding claim 4, Zhang does not disclose the additional limitations. Neither does Selivanov or Li.
Gao discloses: 4. The method of claim 2, further comprising: identifying, by the summary generation engine, an audience for the summary audio file; ("[0153] In order to make the image caption information more adaptable to using requirements of different objects, in a possible implementation, in response to the generated language of the target image caption information being a non-specified language, the computer device can convert the generated caption information in the first language to the caption information in a specified language. For example, the image caption information generated by the information generating model is caption information in English, and the specified language required by the target object is Chinese, then after the information generating model generates the English image caption information, the computer device can translate the English image caption information to Chinese image caption information. …” – In this case, the audience is Chinese speakers. Gao does not specify how the language is determined, but it would be inherent that the alternate language would have to be either input or determined in some way.)
determining, by the summary generation engine, a characteristic of the audience; (In this example, the characteristic is also the ability to read Chinese.)
and customizing, by the summary generation engine, the first summary of the piece of data and the summary audio file based on the characteristic of the audience. ("[0154] A language type of the outputted image caption information, that is, the type of the specified language can be set by the relevant object according to actual requirements." )
Note that the instant application Specification [0047] and [0048] state that identifying an audience and determining a characteristic can simply be an input.
Zhang, Selivanov, Li, and Gao are considered analogous art to the claimed invention because they disclose image captioning systems. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination with translation to another language as taught by Gao. Doing so would have been beneficial to adapt the caption to specific user language requirements. (Gao [0153] )
Regarding claim 5, Zhang does not disclose the additional limitations. Neither does Selivanov or Li.
Gao discloses: 5. The method of claim 4, wherein customizing the first summary of the piece of data and the summary audio file based on the characteristic of the audience further comprises at least one of, based on the characteristic of the audience: reducing or increasing, by the summary generation engine, a comprehensive complexity of diction making up the first summary; shortening or lengthening, by the summary generation engine, a length of the first summary; decreasing or increasing, by the summary generation engine, a playback speed of the summary audio file; changing, by the summary generation engine, a language of the first summary or a playback language of the summary audio file; or redacting or omitting, by the summary generation engine, one or more portions of the first summary or the summary audio file. ("[0153]...in response to the generated language of the target image caption information being a non-specified language, the computer device can convert the generated caption information in the first language to the caption information in a specified language..." )
Rationale for combination as provided for Claim 4.
Regarding claim 6, Zhang does not disclose the additional requirements. Neither does Selivanov or Li.
Gao discloses: 6. The method of claim 5, wherein the characteristic of the audience comprises at least one of an age, a race, a title, a level of education, a primary language, or a privilege or permission level of the audience. ("[0154] A language type of the outputted image caption information, that is, the type of the specified language can be set by the relevant object according to actual requirements." )
Rationale for combination as provided for Claim 4.
Claim 12 is an apparatus claim with limitations corresponding to the limitations of Claim 2 and is rejected under similar rationale.
Claim 13 is an apparatus claim with limitations corresponding to the limitations of Claim 3 and is rejected under similar rationale.
Claim 14 is an apparatus claim with limitations corresponding to the limitations of Claim 4 and is rejected under similar rationale.
Claim 15 is an apparatus claim with limitations corresponding to the limitations of Claim 5 and is rejected under similar rationale
Claim 17 is a computer program product claim with limitations corresponding to the limitations of Claim 2 and is rejected under similar rationale.
Claim 18 is a computer program product claim with limitations corresponding to the limitations of Claim 3 and is rejected under similar rationale.
Claim 19 is a computer program product claim with limitations corresponding to the limitations of Claim 4 and is rejected under similar rationale.
Claim 20 is a computer program product claim with limitations corresponding to the limitations of Claim 5 and is rejected under similar rationale.
Claim(s) 9-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Selivanov and Li as applied in claim 8 above, in further view of Baker et al. (US 20140114643 A1)
Regarding claim 9, Zhang does not disclose the additional limitations. Neither does Selivanov or Li.
Baker discloses: 9. The method of claim 8 wherein: the feedback comprises a request for additional data; (“[0022] Instance 4, of FIG. 2B, shows another image preview 202. Assume that the user presses the camera icon 204 to take the picture which is reflected in instance 5. Instance 5 shows a picture 206 and an autogenerated sentence at 208. In this example, the autogenerated sentence reads “Kailana and a friend at the National Bison Range near Moises, Mont.” Assume that the user wants to identify the other person (e.g., the friend) and as such taps the face on the screen as indicated at 210.” – identifying the other person is a request for additional data.)
identifying, by the feature identification engine, a second actual feature of the piece of the data, wherein the second actual feature is associated with the additional data specified in the request; (“[0024] Instance 7 shows the user entry label of “Simon” now associated with the image. This information can be utilized to update the autogenerated sentence.” – the person’s name is a second actual feature. …)
generating, by the summary generation engine and based on the second actual feature, a second summary of the piece of data, wherein the second summary is different from the first summary; (“[0024] … Instance 8 shows the results where the updated autogenerated sentence indicated at 208 now reads “Kailana and Simon at the National Bison Range near Moises, Mont.”…” – the updated summary includes the second person’s name.)
and causing, by the summary generation engine, display of the second summary. (Fig. 2B shows that the new summary is displayed on a computing device 102.)
Zhang, Selivanov, Li, and Baker are considered analogous art to the claimed invention because they disclose image analysis. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination with an option to request additional detail in the summary as taught by Baker. Doing so would have been beneficial if the user wanted to identify additional people (Baker [0022]) or specify how much detail is included (Baker [0053]).
Regarding claim 10, Zhang discloses that the summary is based on multiple attributes (Fig. 1 shows “person” and “bat”.) Zhang does not specifically disclose a second summary based on an updated piece of data. Neither does Selivanov or Li.
Baker discloses: 10. The method of claim 9, wherein the second summary is also generated based on the actual feature of the piece of data. (“[0024] … Instance 8 shows the results where the updated autogenerated sentence indicated at 208 now reads “Kailana and Simon at the National Bison Range near Moises, Mont.”…” – in this example, Kailana is the first person’s name (first actual feature) and is still included in the summary.)
Rationale for combination as provided for Claim 9.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JON C MEIS whose telephone number is (703)756-1566. The examiner can normally be reached Monday - Thursday, 8:30 am - 5:30 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hai Phan can be reached on 571-272-6338. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JON CHRISTOPHER MEIS/Examiner, Art Unit 2654
/HAI PHAN/Supervisory Patent Examiner, Art Unit 2654
Prosecution Insights