Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This office action is in response to application 19/002,068 filed on December 26, 2024.
Claims 1-20 are pending.
Information Disclosure Statement
As required by M.P.E.P. 609(C), the applicant’s submission of the Information Disclosure Statement dated January 2, 2025 is acknowledged by the examiner and the cited references have been considered in the examination of the claims now pending. As required by M.P.E.P 609, a copy of the PTOL-1449 initialed and dated by the examiner is attached to the office action.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-5, 8-13, 15-17, and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Mansi et al. (US20190333626A1) in further view of Divine et al. (US20180247024A1).
Regarding claim 1, Mansi teaches a system, comprising:
one or more processors, coupled with memory (The virtual agent 210 is a computer-based intelligent artificial agent ... The virtual agent 210 is implemented on one or more computers or processors by executing computer program instructions (code) loaded into memory)([0019]), to:
receive a … prompt from a user (the virtual agent 210 can interact like a virtual assistant, through natural language)([0033]; a user interacts with a virtual assist using prompts);
generate, based at least in part on a plurality of sets of data associated with at least one medical procedure, an output corresponding to the … prompt, wherein each of the plurality of sets of data has a different modality, wherein the plurality of sets of data comprises depth data (data related to the surgical procedure from multiple data sources is fused based on a current context ... at step 302, data related to the surgical procedure is received from multiple data sources ... the data can include medical images ... camera images ... other sensor/actuator data ... At step 304, the data integration middleware 205 integrates the data into a common data model and representation system ... At step 306, the data from the different data sources is fused in a common frame of reference ... depth cameras and sensors can be used ... virtual agent 210 can answer questions (using the real-time reasoning based on the real-time data fusion and quantification))([0006], [0022], [0023], [0026], and [0033]; an answer to a prompt is generated, the answer depends on a plurality of data sets (i.e., sources) including depth data); and
provide the output for display (At step 314, the virtual agent 210 interacts with the operators and the OR environment via the presentation layer 214 to ... automatically present the relevant information according to the reasoning system, etc.)([0032] and [0033]).
Mansi differs from the claim in that Mansi fails to teach receiving a text prompt from a user to generate and display relevant information.
However, receiving a text prompt from a user to generate and display relevant information is taught by Divine (the user can provide input via the user device 102 using a suitable input mechanism (e.g., a keyboard, a keypad, a touch-screen, gesture based input, voice recognition based input, etc.), identifying an object and requesting information about that object … Based on reception of the user input ... the environment assessment component 1006 can determine the answer to the user's question/request)([0150]).
The examiner notes Mansi and Divine teach outputting information in response to a query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi to include the receiving of Divine such that a text prompt is received from a user to generate and display relevant information. One would be motivated to make such a combination to provide the advantage of aiding in the retrieval of information.
Regarding claim 2, Mansi-Divine teach the system of claim 1, wherein the output includes a text response and a visual annotation of video data, the text response includes a number responsive to the text prompt, and the plurality of sets of data comprises the video data (although Mansi discloses of the sets of data comprise video data (collects the real-time data from various real-time data streams ... real-time data streams can include ... videos)([0029]). Mansi does not teach outputting text response and visual annotation, wherein the text includes a number; however, said outputting is taught by Divine (the user may say for instance, “show me supplies that are low in stock,” “show me the equipment having maintenance issues,” “show me the part of this imaging device that is not working properly,” etc. ... the environment assessment component 1006 can determine the answer to the user's question/request ... The feedback component can further generate feedback information that answers the user's question or responds to the user's request . With reference to FIG. 12A ... The text also provides additional information regarding a scheduled maintenance appointment for the imaging device 1202 ... FIG. 15 presents an example AR visualization)([0150], [0154], and [0170]; Figures 12A and 15 - output including text, annotations, and text number is shown). The examiner notes Mansi and Divine teach presenting information in response to a user query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi to include the outputting of Divine such that output includes text response and visual annotation, wherein the text includes a number. One would be motivated to make such a combination to provide the advantage of aiding user understand by providing rich communication).
Regarding claim 4, Mansi-Divine teach the system of claim 1, wherein the output includes a text response and an image, the text response includes at least a partial description of the image and is responsive to the text prompt (although Mansi does not teach outputting text response and an image, wherein the text includes a partial description of the image, said outputting is taught by Divine (the user may say for instance, “show me supplies that are low in stock,” “show me the equipment having maintenance issues,” “show me the part of this imaging device that is not working properly,” etc. ... the environment assessment component 1006 can determine the answer to the user's question/request ... The feedback component can further generate feedback information that answers the user's question or responds to the user's request . With reference to FIG. 12A ... The text also provides additional information regarding a scheduled maintenance appointment for the imaging device 1202 ... FIG. 15 presents an example AR visualization)([0150], [0154], and [0170]; Figures 12A and 15 - output including text and an image such that text partially describes the image is shown). The examiner notes Mansi and Divine teach presenting information in response to a user query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi to include the outputting of Divine such that output includes text response and an image, wherein the text includes a partial description of the image. One would be motivated to make such a combination to provide the advantage of aiding user understand by providing rich communication).
Regarding claim 5, Mansi-Divine teach the system of claim 4, wherein the partial description of the image includes a description of at least one of an absolute location or a relative location within the medical environment of a first object in the plurality of sets of data, the relative location of the first object relative to a second object in the medical environment; or the partial description of the image includes a description of a layout of one or more objects in the plurality of sets of data, the image corresponds to a 3D reconstruction of a medical environment based on point cloud data, the medical environment corresponds to the medical procedure, and the plurality of set of data includes the point cloud data (Divine - the user can pose a question asking for information about a location or room in a healthcare facility such as requesting information about where certain supplies are located in the room ... provide the user with feedback in the form of an overlay that notes the particular location in the room including the non-latex gloves. For example, if the non-latex gloves are located on the left side second shelf of the supply cabinet, the overlay data can highlight, mark, or otherwise draw attention to the left side second shelf of the supply cabinet)([0150] and [0151]).
Regarding claim 8, Mansi-Divine teach the system of claim 1, wherein the output includes a text response and a data visualization output, the text response includes at least a partial description of the data visualization output (although Mansi does not teach outputting text response and data visualization, wherein the text includes a partial description of the data visualization, said outputting is taught by Divine (the user may say for instance, “show me supplies that are low in stock,” “show me the equipment having maintenance issues,” “show me the part of this imaging device that is not working properly,” etc. ... the environment assessment component 1006 can determine the answer to the user's question/request ... The feedback component can further generate feedback information that answers the user's question or responds to the user's request . With reference to FIG. 12A ... The text also provides additional information regarding a scheduled maintenance appointment for the imaging device 1202 ... FIG. 15 presents an example AR visualization)([0150], [0154], and [0170]; Figures 12A and 15 - output including text and data visualization such that text partially describes the data visualization is shown). The examiner notes Mansi and Divine teach presenting information in response to a user query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi to include the outputting of Divine such that output includes text response and data visualization, wherein the text includes a partial description of the data visualization. One would be motivated to make such a combination to provide the advantage of aiding user understand by providing rich communication).
Regarding claim 9, Mansi-Divine teach the system of claim 1, the processors to: determine a modality that is responsive to the text prompt, wherein the modality includes at least one of video data, an annotation of video data, an image, and a data visualization (Divine - The feedback component can further generate feedback information that answers the user's question or responds to the user's request)([0150]; an appropriate modality of output is generated to answer a user’s question).
Regarding claim 10, Mansi-Divine teach the system of claim 1, the processors to:
extract one or more features from one of more of the plurality of sets of data; and generate one or more fused features each including one or more of the features each having the different modality; and generate the output based on one or more of the fused features (Mansi - At step 304, the data integration middleware 205 integrates the data into a common data model and representation system ... At step 306, the data from the different data sources is fused in a common frame of reference ... At step 314, the virtual agent 210 interacts with the operators and the OR environment via the presentation layer 214 to ... automatically present the relevant information according to the reasoning system, etc.)([0022], [0023], [0032] and [0033]; meaningful representations of data are extracted and represented in a model, the representations are fused, and relevant output is generated).
Regarding claim 11, Mansi-Divine teach the system of claim 1, the processors to:
select, according to a determination that the plurality of sets of data includes video data, a neural network configured to extract features from the video data; and extract, by the neural network, the one or more features from the video data (Mansi - artificial intelligence is used to achieve the image registration ... task-specific intelligent artificial agents are trained to perform various registration tasks corresponding to anatomies being registered and the medical imaging modalities of the images being registered ... the task-specific intelligent artificial agents can utilize trained deep neural networks ... includes detecting and consistently tracking landmarks in medical images and video)([0024] and [0028]; a specific agent trained neural network is used to extract (i.e., register) features in video).
Regarding claim 12, Mansi a system, comprising:
one or more processors, coupled with memory (The virtual agent 210 is a computer-based intelligent artificial agent ... The virtual agent 210 is implemented on one or more computers or processors by executing computer program instructions (code) loaded into memory)([0019]), to:
receive multi-modal data comprising video data, analytics data, and metadata for one or more medical procedures each to update one or more models (data related to the surgical procedure from multiple data sources is fused based on a current context ... at step 302, data related to the surgical procedure is received from multiple data sources ... the data can include medical images ... camera images ... other sensor/actuator data ... At step 304, the data integration middleware 205 integrates the data into a common data model and representation system ... At step 306, the data from the different data sources is fused in a common frame of reference)([0006], [0022], and [0023]; data including video, analytics data (e.g., medical image), and metadata (e.g., sensor data) is input into a model);
generate, using a first model configured to detect image features, a first feature that identifies an object in the video data for a plurality of medical procedures, wherein the video includes at least one of medical staff, a patient, a robotic system or instrument, or medical environment (artificial intelligence is used to achieve the image registration ... task-specific intelligent artificial agents are trained to perform various registration tasks corresponding to anatomies being registered and the medical imaging modalities of the images being registered ... the task-specific intelligent artificial agents can utilize trained deep neural networks ... includes detecting and consistently tracking landmarks in medical images and video)([0024] and [0028]; an agent is used to register features in video of a medical environment);
generate, using a second model, a second feature that identifies features in a … prompt (the virtual agent 210 can interact like a virtual assistant, through natural language)([0033]; a user interacts with a virtual assist using prompts, prompts are processed using a natural language model to determine features);
generate, by a third model and based on the first feature and the second feature, an output responsive to the input … prompt, wherein the output comprises at least one of text or media content (At step 314, the virtual agent 210 interacts with the operators and the OR environment via the presentation layer 214 to ... visualize the medical images and other data ... automatically present the relevant information according to the reasoning system, etc.)([0032] and [0033]; a virtual agent generates output based on registered features and natural language features).
Mansi differs from the claim in that Mansi fails to teach receiving a text prompt from a user to generate and display relevant information, determining a loss with respect to output (i.e., new data), and updating a model based on the new data.
However, receiving a text prompt from a user to generate and display relevant information, determining new data, and updating a model based on the new data is taught by Divine (Machine learning is a type of artificial intelligence (AI) that provides computers with the ability to learn without being explicitly programmed ... Supervised algorithms can apply what has been learned in the past to new data … the user can provide input via the user device 102 using a suitable input mechanism (e.g., a keyboard, a keypad, a touch-screen, gesture based input, voice recognition based input, etc.), identifying an object and requesting information about that object … Based on reception of the user input ... the environment assessment component 1006 can determine the answer to the user's question/request)([0073] and [0150]).
The examiner notes Mansi and Divine teach outputting information in response to a query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi to include the receiving of Divine such that a text prompt is received from a user to generate and display relevant information, loss with respect to output is determined, and a model is updated based on the loss. One would be motivated to make such a combination to provide the advantage of aiding in the retrieval of information.
Regarding claim 13, Mansi-Divine teach the system of claim 12, wherein the output includes a text response and a visual annotation of the video data, and the text response includes a number responsive to the text prompt (although Mansi discloses of the sets of data comprise video data (collects the real-time data from various real-time data streams ... real-time data streams can include ... videos)([0029]). Mansi does not teach outputting text response and visual annotation, wherein the text includes a number; however, said outputting is taught by Divine (the user may say for instance, “show me supplies that are low in stock,” “show me the equipment having maintenance issues,” “show me the part of this imaging device that is not working properly,” etc. ... the environment assessment component 1006 can determine the answer to the user's question/request ... The feedback component can further generate feedback information that answers the user's question or responds to the user's request . With reference to FIG. 12A ... The text also provides additional information regarding a scheduled maintenance appointment for the imaging device 1202 ... FIG. 15 presents an example AR visualization)([0150], [0154], and [0170]; Figures 12A and 15 - output including text, annotations, and text number is shown). The examiner notes Mansi and Divine teach presenting information in response to a user query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi to include the outputting of Divine such that output includes text response and visual annotation, wherein the text includes a number. One would be motivated to make such a combination to provide the advantage of aiding user understand by providing rich communication).
Regarding claim 15, Mansi-Divine teach the system of claim 12, wherein the output includes a text response and an image, the text response includes at least a partial description of the image and is responsive to the text prompt, and the media content comprises the image (although Mansi does not teach outputting text response and an image, wherein the text includes a partial description of the image, said outputting is taught by Divine (the user may say for instance, “show me supplies that are low in stock,” “show me the equipment having maintenance issues,” “show me the part of this imaging device that is not working properly,” etc. ... the environment assessment component 1006 can determine the answer to the user's question/request ... The feedback component can further generate feedback information that answers the user's question or responds to the user's request . With reference to FIG. 12A ... The text also provides additional information regarding a scheduled maintenance appointment for the imaging device 1202 ... FIG. 15 presents an example AR visualization)([0150], [0154], and [0170]; Figures 12A and 15 - output including text and an image such that text partially describes the image is shown). The examiner notes Mansi and Divine teach presenting information in response to a user query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi to include the outputting of Divine such that output includes text response and an image, wherein the text includes a partial description of the image. One would be motivated to make such a combination to provide the advantage of aiding user understand by providing rich communication).
Regarding claim 16, Mansi-Divine teach the system of claim 15, wherein the partial description of the image includes a description of a location of an object in the multi-modal data (Divine - the user can pose a question asking for information about a location or room in a healthcare facility such as requesting information about where certain supplies are located in the room ... provide the user with feedback in the form of an overlay that notes the particular location in the room including the non-latex gloves. For example, if the non-latex gloves are located on the left side second shelf of the supply cabinet, the overlay data can highlight, mark, or otherwise draw attention to the left side second shelf of the supply cabinet)([0150] and [0151]).
Regarding claim 17, Mansi-Divine teach the system of claim 15, wherein the partial description of the image includes a description of a layout of one or more objects in the plurality of sets of data, and the image is based on the metadata (Divine - the user can pose a question asking for information about a location or room in a healthcare facility such as requesting information about where certain supplies are located in the room ... provide the user with feedback in the form of an overlay that notes the particular location in the room including the non-latex gloves. For example, if the non-latex gloves are located on the left side second shelf of the supply cabinet, the overlay data can highlight, mark, or otherwise draw attention to the left side second shelf of the supply cabinet)([0150] and [0151]).
Regarding claim 19, Mansi-Divine teach the system of claim 12, wherein the output includes a text response and a data visualization output, and the text response includes at least a partial description of the data visualization output (although Mansi does not teach outputting text response and data visualization, wherein the text includes a partial description of the data visualization, said outputting is taught by Divine (the user may say for instance, “show me supplies that are low in stock,” “show me the equipment having maintenance issues,” “show me the part of this imaging device that is not working properly,” etc. ... the environment assessment component 1006 can determine the answer to the user's question/request ... The feedback component can further generate feedback information that answers the user's question or responds to the user's request . With reference to FIG. 12A ... The text also provides additional information regarding a scheduled maintenance appointment for the imaging device 1202 ... FIG. 15 presents an example AR visualization)([0150], [0154], and [0170]; Figures 12A and 15 - output including text and data visualization such that text partially describes the data visualization is shown). The examiner notes Mansi and Divine teach presenting information in response to a user query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi to include the outputting of Divine such that output includes text response and data visualization, wherein the text includes a partial description of the data visualization. One would be motivated to make such a combination to provide the advantage of aiding user understand by providing rich communication).
Regarding claim 20, Mansi teach a system, comprising: one or more processors, coupled with memory, to:
receive a … prompt from a user (the virtual agent 210 can interact like a virtual assistant)([0019]; user interacts with a virtual assist using prompts);
determine a … prompt feature for the … prompt (through natural language)([0033]; prompts are processed using a natural language model to determine features);
identify, based at least in part on the … prompt feature, a plurality of sets of data associated with at least one medical procedure, wherein each of the plurality of sets of data has a different modality, wherein at least one set of the plurality of sets of data comprises depth data (data related to the surgical procedure from multiple data sources is fused based on a current context ... at step 302, data related to the surgical procedure is received from multiple data sources ... the data can include medical images ... camera images ... other sensor/actuator data ... At step 304, the data integration middleware 205 integrates the data into a common data model and representation system ... At step 306, the data from the different data sources is fused in a common frame of reference ... depth cameras and sensors can be used ... virtual agent 210 can answer questions (using the real-time reasoning based on the real-time data fusion and quantification))([0006], [0022], [0023], [0026], and [0033]; an answer to a prompt is generated, the answer depends on a plurality of data sets (i.e., sources) including depth data); and
generate, based on the processing of the plurality of sets of data, an output responsive to the … prompt feature (At step 314, the virtual agent 210 interacts with the operators and the OR environment via the presentation layer 214 to ... automatically present the relevant information according to the reasoning system, etc.)([0032] and [0033]).
Mansi differs from the claim in that Mansi fails to teach receiving a text prompt from a user to generate and display relevant information by processing data based on the text prompt. However, receiving a text prompt from a user to generate and display relevant information by processing data based on the text prompt is taught by Divine (the user can provide input via the user device 102 using a suitable input mechanism (e.g., a keyboard, a keypad, a touch-screen, gesture based input, voice recognition based input, etc.), identifying an object and requesting information about that object … the user can pose a question asking for information about a location or room in a healthcare facility such as requesting information about where certain supplies are located in the room … Based on reception of the user input ... the environment assessment component 1006 can determine the answer to the user's question/request … For example, if the non-latex gloves are located on the left side second shelf of the supply cabinet, the overlay data can highlight, mark, or otherwise draw attention to the left side second shelf of the supply cabinet)([0150] and [0151]).
The examiner notes Mansi and Divine teach outputting information in response to a query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi to include the receiving of Divine such that a text prompt is received from a user to generate and display relevant information by processing data based on the text prompt. One would be motivated to make such a combination to provide the advantage of aiding in the retrieval of information by providing additional input types.
Claims 3 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Mansi, Divine, and in further view of Yous et al. (US20210191978A1).
Regarding claim 3, Mansi-Divine teach the system as applied above, wherein the output includes a text response (although Mansi does not teach outputting text; said outputting is taught by Divine (the user may say for instance, “show me supplies that are low in stock,” “show me the equipment having maintenance issues,” “show me the part of this imaging device that is not working properly,” etc. ... the environment assessment component 1006 can determine the answer to the user's question/request ... The feedback component can further generate feedback information that answers the user's question or responds to the user's request . With reference to FIG. 12A ... The text also provides additional information regarding a scheduled maintenance appointment for the imaging device 1202 ... FIG. 15 presents an example AR visualization)([0150], [0154], and [0170]; Figures 12A and 15 - output including text is shown). The examiner notes Mansi and Divine teach presenting information in response to a user query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi to include the outputting of Divine such that output includes text response. One would be motivated to make such a combination to provide the advantage of aiding user understand by providing rich communication) and the plurality of sets of data comprises the video data (Mansi - (collects the real-time data from various real-time data streams ... real-time data streams can include ... videos)([0029]).
Mansi-Divine differs from the claim in that Mansi-Divine fails to teach output includes a portion of video within a time period.
However, outputting a portion of video within a time period in response to a user prompt is taught by Yous (The search engine 18 generates a search result 110 including, for example, videos, excerpts of the videos ... matching the video query 50 ... Time periods within the video may be determined from time encoding in the video itself, and is evaluated to ensure that the temporal relationship video query term is met ... The video query may be received from a user … At 214, the search engine 18 generates a search result)([0029] and [0030]).
The examiner notes Mansi, Divine, and Yous teach outputting information in response to a query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi-Divine to include the outputting of Yous such that a portion of video within a time period is outputted. One would be motivated to make such a combination to provide the advantage of outputting temporally relevant video.
Regarding claim 14, Mansi-Divine teach the system as applied above, wherein the output includes a text response (although Mansi does not teach outputting text; said outputting is taught by Divine (the user may say for instance, “show me supplies that are low in stock,” “show me the equipment having maintenance issues,” “show me the part of this imaging device that is not working properly,” etc. ... the environment assessment component 1006 can determine the answer to the user's question/request ... The feedback component can further generate feedback information that answers the user's question or responds to the user's request . With reference to FIG. 12A ... The text also provides additional information regarding a scheduled maintenance appointment for the imaging device 1202 ... FIG. 15 presents an example AR visualization)([0150], [0154], and [0170]; Figures 12A and 15 - output including text is shown). The examiner notes Mansi and Divine teach presenting information in response to a user query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi to include the outputting of Divine such that output includes text response. One would be motivated to make such a combination to provide the advantage of aiding user understand by providing rich communication) and the plurality of sets of data comprises the video data (Mansi - (collects the real-time data from various real-time data streams ... real-time data streams can include ... videos)([0029]).
Mansi-Divine differs from the claim in that Mansi-Divine fails to teach output includes a portion of video within a time period.
However, outputting a portion of video within a time period in response to a user prompt is taught by Yous (The search engine 18 generates a search result 110 including, for example, videos, excerpts of the videos ... matching the video query 50 ... Time periods within the video may be determined from time encoding in the video itself, and is evaluated to ensure that the temporal relationship video query term is met ... The video query may be received from a user … At 214, the search engine 18 generates a search result)([0029] and [0030]).
The examiner notes Mansi, Divine, and Yous teach outputting information in response to a query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi-Divine to include the outputting of Yous such that a portion of video within a time period is outputted. One would be motivated to make such a combination to provide the advantage of outputting temporally relevant video.
Claims 6 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Mansi, Divine, and in further view of Chandratillake et al. (US20080086688A1).
Regarding claim 6, Mansi-Divine teach the system as applied above, wherein output include text response which includes a number (although Mansi does not teach outputting text response, wherein the text includes a number; however, said outputting is taught by Divine (the user may say for instance, “show me supplies that are low in stock,” “show me the equipment having maintenance issues,” “show me the part of this imaging device that is not working properly,” etc. ... the environment assessment component 1006 can determine the answer to the user's question/request ... The feedback component can further generate feedback information that answers the user's question or responds to the user's request . With reference to FIG. 12A ... The text also provides additional information regarding a scheduled maintenance appointment for the imaging device 1202 ... FIG. 15 presents an example AR visualization)([0150], [0154], and [0170]; Figures 12A and 15 - output including text including a number is shown). The examiner notes Mansi and Divine teach presenting information in response to a user query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi to include the outputting of Divine such that output includes text response, wherein the text includes a number. One would be motivated to make such a combination to provide the advantage of aiding user understand by providing rich communication).
Mansi-Divine differs from the claim in that outputting a number of portions of video data corresponding to a number response (i.e., number of results) to text prompt.
However, outputting a number of portions of video data corresponding to a number of results to text prompt is taught by Chandratillake (FIGS. 3 a and 3 b illustrate an embodiment of a graphic user interface presented by the search engine to the user based on ... supplied query terms)([0043]; Figure 3A – displaying a number of videos corresponding to a number of results (i.e., 10,000) in response to a text prompt (i.e., “harry potter” query) is shown).
The examiner notes Mansi, Divine, and Chandratillake teach outputting information in response to a query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi-Divine to include the outputting of Chandratillake such that a number of portions of video data corresponding to a number response to text prompt is outputted. One would be motivated to make such a combination to provide the advantage of informing a user of number of relevant videos.
Regarding claim 18, Mansi-Divine teach the system as applied above, wherein output include text response which includes a number (although Mansi does not teach outputting text response, wherein the text includes a number; however, said outputting is taught by Divine (the user may say for instance, “show me supplies that are low in stock,” “show me the equipment having maintenance issues,” “show me the part of this imaging device that is not working properly,” etc. ... the environment assessment component 1006 can determine the answer to the user's question/request ... The feedback component can further generate feedback information that answers the user's question or responds to the user's request . With reference to FIG. 12A ... The text also provides additional information regarding a scheduled maintenance appointment for the imaging device 1202 ... FIG. 15 presents an example AR visualization)([0150], [0154], and [0170]; Figures 12A and 15 - output including text including a number is shown). The examiner notes Mansi and Divine teach presenting information in response to a user query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi to include the outputting of Divine such that output includes text response, wherein the text includes a number. One would be motivated to make such a combination to provide the advantage of aiding user understand by providing rich communication).
Mansi-Divine differs from the claim in that outputting a number of portions of video data corresponding to a number response (i.e., number of results) to text prompt.
However, outputting a number of portions of video data corresponding to a number of results to text prompt is taught by Chandratillake (FIGS. 3 a and 3 b illustrate an embodiment of a graphic user interface presented by the search engine to the user based on ... supplied query terms)([0043]; Figure 3A – displaying a number of videos corresponding to a number of results (i.e., 10,000) in response to a text prompt (i.e., “harry potter” query) is shown).
The examiner notes Mansi, Divine, and Chandratillake teach outputting information in response to a query. As such, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify Mansi-Divine to include the outputting of Chandratillake such that a number of portions of video data corresponding to a number response to text prompt is outputted. One would be motivated to make such a combination to provide the advantage of informing a user of number of relevant videos.
Allowable Subject Matter
Claim 7 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record on form PTO-892 and not relied upon is considered pertinent to applicant's disclosure. Applicant is required under 37 C.F.R. § 1.111(c) to consider the reference fully when responding to this action.
The document cited therein and enumerated below teaches a method and apparatus for providing output of data associated with a medical procedure.
US20180233228A1
US20200237452A1
US20230017202A1
US10600510B2
US11515030B2
WO2025122673A1
ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model
Integrating vision processing and natural language processing with a clinical application
Text Promptable Surgical Instrument Segmentation with Vision-Language Models
Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Training
The document cited therein and enumerated below teaches a method and apparatus for providing output of data in response to prompt input.
US20140355823A1
US20160378861A1
US20180121603A1
US20190147346A1
US20230169363A1
US20230252344A1
US6779060B1
US11087889B2
US11599749B1
US11810351B2
US11803710B1
US11823442B2
CN116628150A
CN116821297A
Exploring Models and Data for Image Question Answering
Teaching machines to reason about what they see
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Yongjia Pan whose telephone number is (571)270-1177. The examiner can normally be reached Monday - Friday, 9:00 AM - 5:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Scott Baderman can be reached at 571-272-3644. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YONGJIA PAN/
Primary Examiner, Art Unit 2118
Prosecution Insights