CTNF 18/944,836 CTNF 84571 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. 07-30-03-h AIA Claim Interpretation 07-30-03 AIA The following is a quotation of 35 U.S.C. 112(f): (f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof. The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph: An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof. 07-30-05 The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked. As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph: (A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function; (B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and (C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function. Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function. Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function. Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. 07-30-06 This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: artificial intelligence model in claim 1. Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof. If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. Claim Rejections - 35 USC § 101 07-04-01 AIA 07-04 35 U.S.C. 101 reads as follows: Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title. Claims 1-9 and 11-12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because claim 1 is directed to a method with the steps comprising receiving, receiving, providing, determining and generating which are nothing more than software instructions. Software instructions are non-statutory under 35 U.S.C. 101. Claims 2-9 and 11-12 depend from claim 1 and comprise further steps, for example claim 2 comprises the step of outputting, therefore claims 2-9 and 11-12 have the same problem as claim 1 and are rejected under the same rationale. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim (s) 1-4, 6, 8, 13-15, 17 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over CAI et. al., “Making Large Multimodal Models Understand Arbitrary Visual Prompts.” arXiv, Dec. 1, 2023, arXiv: 2312.00784v1 [cs.CV] in view of ENS et. al., “Spatial Analytic Interfaces: Spatial User Interfaces for In Situ Visual Analytics,” in IEEE Computer Graphics and Applications , vol. 37, no. 2, pp. 66-79, Mar.-Apr. 2017, doi: 10.1109/MCG.2016.38 . Regarding claim 1, CAI teaches: 1. A method, comprising: receiving, at one or more processors through an interface displaying a first offering of objects, user input selecting a window size and shape, such that the window encompasses one or more objects depicted on the interface (CAI: pg. 9, Table 6, image having 3 bounding boxes (i.e., window having size and shape); pg. 18 fig. 16(a) image having vertical rectangles (i.e., window having size and shape)) ; receiving, at the one or more processors, a second mode of input (CAI: pg. 9, Table 6, User text prompt, “Between Object 1: the object within the red mask contour, Object 2: the object…”) ; providing the one or more objects and the second mode of input as a combined input to an artificial intelligence model (CAI: pg. 9, Table 6, User text prompt, “Between Object 1: the object within the red mask contour, Object 2: the object…,” see also caption of fig. 1: “We directly overlay diverse visual prompts (e.g., arrows, boxes, circles, scribbles) onto the original image, and then feed the corresponding visual features along with text embeddings into the large multimodal model”) ; determining, by the artificial intelligence model, at least one relationship between the one or more objects and the second mode of input (CAI: pg. 9, Table 6, User text prompt, “Between Object 1: the object within the red mask contour, Object 2: the object within the blue mask contour, and Object 3: the object within the green mask contour, which one has something on top of it?”) ; and generating, by the artificial intelligence model, a response based on the at least one relationship (CAI: pg. 9, Table 6, ViP-LlaVA-7B “Object 2: the object within the blue mask contour has something on top of it.” or ViP-LlaVA-13B “Object 2: the object within the blue mask contour has something on top of it.”) . CAI doesn’t teach however the analogous prior art ENS teaches: the interface is a spatial interface (ENS: pg. 71, Augmentation section, right col., 2 nd par. “On physical displays, visual links…” – pg. 72 left col., 1 st par.) . It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the interface is a spatial interface as shown in ENS with CAI for the benefit of allowing the user actually move among and between the visualizations and links to gain the best perspective without the need for abstract virtual navigation [ENS: pg. 71, right col., Augmentation section, 2 nd par., line 7-pg. 72, left col. line 3]. Regarding claim 2, CAI teaches: 2. The method of claim 1, comprising outputting the response visually through the interface (CAI: pg. 7, fig. 7, Robot face responds with: “The ellipse with the thick contour encircles the girl.”) . CAI doesn’t teach however the analogous prior art ENS (with the same motivation of claim 1) teaches: the interface is a spatial interface ((ENS: pg. 71, Augmentation section, right col., 2 nd par. “On physical displays, visual links…” – pg. 72 left col., 1 st par.)) . Regarding claim 3, CAI teaches: 3. The method of claim 1, wherein the one or more objects comprise a first object of a first type and a second object of a second type different from the first type (CAI: pg. 7, fig. 7, The 1 st object type is a boy and the 2 nd object type is a girl) . Regarding claim 4, CAI teaches: 4. The method of claim 3, wherein the first object comprises an image (CAI: pg. 7, fig. 7, The 1 st object is an image of a boy) . Regarding claim 6, CAI teaches: 6. The method of claim 1, wherein the second mode of input comprises text or verbal input (CAI: pg. 9, Table 6, text prompt of user: “Between Object 1: the object within the red mask contour, Object 2: the object within the blue mask contour, and Object 3: the object within the green mask contour, which one has something on top of it?”) . Regarding claim 8, CAI teaches: 8. The method of claim 1, wherein the second mode of input is received through a second interface separate from the interface (CAI: pg. 7, fig. 7 Cartoon face of a boy asks: “Among the ellipses, one with a thick contour and one with a thin contour, which one encircles the girl?” which is separate from the interface (i.e., image of a boy and girl)) . CAI doesn’t teach however the analogous prior art ENS (with the same motivation of claim 1) teaches: the interface is a spatial interface (ENS: pg. 71, Augmentation section, right col., 2 nd par. “On physical displays, visual links…” – pg. 72 left col., 1 st par.) . Claim 13 is analogous to claim 1 and is therefore rejected using the same rationale. Claim 13 further requires a different preamble and additional limitations also taught by CAI : A system, comprising: memory; and one or more processors in communication with the memory (CAI: pg. 4, right col., lines 10-15; POSITA would recognize that a method that feeds images into GPT-4V comprises a system, memory and one or more processors in communication with the memory) . Claim 14 is analogous to claim 2 and is therefore rejected using the same rationale. Regarding claim 15, CAI teaches: 15. The system of claim 13, wherein the one or more objects comprise at least one image (CAI: pg. 7, fig. 7, The boy and girl (i.e., objects) comprise an image) . Claim 17 is analogous to claim 6 and is therefore rejected using the same rationale. Claim 20 is analogous to claim 1 and is therefore rejected using the same rationale. Claim 20 further requires a different preamble also taught by CAI : A non-transitory computer-readable medium storing instructions executable by one or more processors for performing a method (CAI: pg. 4, right col., lines 10-15; POSITA would recognize that a method that feeds images into GPT-4V comprises a non-transitory computer-readable medium storing instructions executable by one or more processors) . 07-21-aia AIA Claim (s) 5 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over CAI in view of ENS in view of PURCELL (US8,605,102B1) . Regarding claim 5, the previous combination of CAI and ENS don’t teach however the analogous prior art PURCELL teaches: 5. The method of claim 1, further comprising rasterizing the one or more objects encompassed in the window (PURCELL: col. 11 lines 24-35) . It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine rasterizing the one or more objects encompassed in the window as shown in PURCELL with the previous combination for the benefit of allowing memory access operations performed when processing those fine raster tiles (e.g., texture fetches, etc.) to be localized to a single frame buffer bank corresponding to the screen region. Localizing memory access operations in this manner may improve the throughput of PPU 202 [PURCELL, col. 13 lines 46-52]. Claim 16 is analogous to claim 5 and is therefore rejected using the same rationale . 07-21-aia AIA Claim (s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over CAI in view of ENS in view of LANDING AI , “Landing AI Launches the World's First Commercial Visual Prompting Capability in LandingLens,” APR 25, 2023, Landing AI Launches the World's First Commercial Visual Prompting Capability in LandingLens in view of BESANCON , et. al., “The State of the Art of Spatial Interfaces for 3D Visualization,” COMPUTER GRAPHICS forum Volume 40 (2021), number 1 pp. 293–326 . Regarding claim 7, the previous combination of CAI and ENS don’t teach however the analogous prior art LANDING AI teaches: 7. The method of claim 1, wherein the second mode of input is received through the interface (LANDING AI: pg. 2 lines 5-14 and image of hummingbird; Visual prompting is considered as the second mode; the image of the humming bird is considered as the interface) . It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine wherein the second mode of input is received through the interface as shown in LANDING AI with the previous combination for the benefit of significantly reducing the time to get a computer vision result from days or months to minutes or even seconds [LANDING AI, pg. 2 lines 4-6]. LANDING AI doesn’t teach however the analogous prior art BESANCON teaches: input is received through a spatial interface (BESANCON: pg. 294, right col., 2 nd par., lines 1-7) . It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine input is received through a spatial interface as shown in BESANCON with the previous combination for the benefit of providing several theoretical advantages: tangible interaction mimics everyday interaction with the real-world [Fit96, BIAI17b], mid-air gestures enable hands-free interaction with medical data during surgeries, and hybrid interaction leverages the benefits of multiple interaction paradigms [BESANCON pg. 294, right col., 2 nd par., lines 7-12] . 07-21-aia AIA Claim (s) 9 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over CAI in view of ENS in view of INAYOSHI et. al., “Bounding-Box Channels for Visual Relationship Detection,” In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12350. Springer, Cham . Regarding claim 9, the previous combination of CAI and ENS don’t teach however the analogous prior art INAYOSHI teaches: 9. The method of claim 1, wherein the window encompasses multiple objects, and wherein determining the at least one relationship comprises identifying a relationship among the multiple objects (INAYOSHI: pg. 689, fig. 4 and caption, “…visual relationship detection in the three tasks proposed in [13]: predicate detection, phrase detection and relationship detection. For predicate detection, classes and bounding boxes of objects are given in addition to an image, and the output is the predicate”) . It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine wherein the window encompasses multiple objects, and wherein determining the at least one relationship comprises identifying a relationship among the multiple objects as shown in INAYOSHI with the previous combination for the benefit of improving the accuracy of VRD (visual relationship detection) [INAYOSHI, pg. 683 last two lines]. Claim 18 is analogous to claim 9 and is therefore rejected using the same rationale . 07-21-aia AIA Claim (s) 10 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over CAI in view of ENS in view of SILFVERBERG (US2003/0043114A1) in view of BESANCON . Regarding claim 10, the previous combination of CAI and ENS don’t teach however the analogous prior art SILFVERBERG teaches: 10. The method of claim 1, further comprising: receiving through the interface a manipulation input (SILFVERBERG: fig. 2 see also par. 24) ; and adjusting, by the one or more processors, the interface in response to the manipulation input, the adjusting comprising panning or zooming the interface to display a second offering of objects different from the first offering of objects (SILFVERBERG: par. 25, lines 12-19) . It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine receiving through the interface a manipulation input; and adjusting, by the one or more processors, the interface in response to the manipulation input, the adjusting comprising panning or zooming the interface to display a second offering of objects different from the first offering of objects as shown in SILFVERBERG with the previous combination for the benefit of addressing a shortcoming in the art by developing a method and device to overcome the problems in manipulating documents and data files that are larger than the size of a display on a data processing device [SILFVERBERG par. 7]. The previous combination of CAI in view of ENS and SILFVERBERG doesn’t teach however the analogous prior art BESANCON teaches: the interface is a spatial interface (BESANCON: pg. 294, right col., 2 nd par., lines 1-5) . It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the interface is a spatial interface as shown in BESANCON with the previous combination for the benefit of providing several theoretical advantages: tangible interaction mimics everyday interaction with the real-world [Fit96, BIAI17b], mid-air gestures enable hands-free interaction with medical data during surgeries, and hybrid interaction leverages the benefits of multiple interaction paradigms [BESANCON pg. 294, right col., 2 nd par., lines 7-12]. Claim 19 is analogous to claim 10 and is therefore rejected using the same rationale . 07-21-aia AIA Claim (s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over CAI in view of ENS in view of NIELSEN , “Direct Manipulation: A Fundamental Element of Graphical User Interfaces” JUL 27, 2023 , Direct Manipulation: A Fundamental Element of Graphical User Interfaces Regarding claim 11, the previous combination of CAI and ENS remains as above but doesn’t teach however the analogous prior art NIELSEN teaches: 11. The method of claim 1, further comprising receiving input adjusting a location of a first object of the first offering of objects relative to a second object of the first offering of objects (NIELSEN: pg. 2, lines 4-6, 14-18, pg. 3, Downsides of Direct Manipulation, lines 1-3) . It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine receiving input adjusting a location of a first object of the first offering of objects relative to a second object of the first offering of objects as shown in NIELSEN with the previous combination for the benefit of reducing cognitive load of users who interact with visual on-screen objects [NIELSEN, see summary] . 07-21-aia AIA Claim (s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over CAI in view of ENS in view of TERNULF (US7,071,952B1) . Regarding claim 12, the previous combination of CAI and ENS remains as above but doesn’t teach however the analogous prior art TERNULF teaches: 12. The method of claim 1, further comprising adding an object to the first offering of objects ( TERNULF : col. 2 lines 14-33) . It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine adding an object to the first offering of objects as shown in TERNULF with the previous combination for the benefit of addressing a shortcoming in the prior art in that when inserting objects of different types into a working area is the fact that the cursor always needs to be moved between the icons or menus from which the object type is chosen, and the desired location [TERNULF, col. 1 line 66-col. 2 line 3]. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to MAURICE L MCDOWELL, JR whose telephone number is (571)270-3707. The examiner can normally be reached Mon-Fri: 2pm-10pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said A. Broome can be reached at 571-272-2931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /MAURICE L. MCDOWELL, JR/Primary Examiner, Art Unit 2612 Application/Control Number: 18/944,836 Page 2 Art Unit: 2612 Application/Control Number: 18/944,836 Page 3 Art Unit: 2612 Application/Control Number: 18/944,836 Page 4 Art Unit: 2612 Application/Control Number: 18/944,836 Page 5 Art Unit: 2612 Application/Control Number: 18/944,836 Page 6 Art Unit: 2612 Application/Control Number: 18/944,836 Page 7 Art Unit: 2612 Application/Control Number: 18/944,836 Page 8 Art Unit: 2612 Application/Control Number: 18/944,836 Page 9 Art Unit: 2612 Application/Control Number: 18/944,836 Page 10 Art Unit: 2612 Application/Control Number: 18/944,836 Page 11 Art Unit: 2612 Application/Control Number: 18/944,836 Page 12 Art Unit: 2612 Application/Control Number: 18/944,836 Page 13 Art Unit: 2612 Application/Control Number: 18/944,836 Page 14 Art Unit: 2612 Application/Control Number: 18/944,836 Page 15 Art Unit: 2612 Application/Control Number: 18/944,836 Page 16 Art Unit: 2612
Prosecution Insights