Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 19 is objected to because of the following informalities: claim 19 is listed as being dependent on claim 10, which is a method claim. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 9, and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kazansky et al. (U.S. PGPUB 20170364153) in view of Brown et al. (U.S. PGPUB 20230306690) and further in view of Osborn et al. (U.S. PGPUB 20170221276).
With respect to claim 1, Kazansky et al. disclose an electronic device (paragraph 53, FIG. 2 is a block diagram illustrating modules (e.g., components) of the HMD 101) comprising: at least one camera (paragraph 60, the sensors 202 may include an external camera 302); a display (paragraph 53, a display 204); memory storing instructions (paragraph 106, The disk drive unit 1316 includes a computer-readable medium 1322 on which is stored one or more sets of data structures and instructions 1324); and at least one processor, comprising processing circuitry, electrically coupled to the at least one camera and the display (paragraph 55, The processor 212 may include an AR application 214 and an interference status unit 216…In one example embodiment, the AR application 214 generates a visualization of information related to the objects A 116, B 118 when the HMD 101 captures an image of the objects A 116), wherein the instructions cause, when being executed by at least one processor, individually and/or collectively, is configured to:
acquire an image using the at least one camera (paragraph 45, The user 102 may point a camera of the HMD 101 to capture an image of the objects A 116 and B 118 in the physical environment 114);
based on the image being acquired, identify at least one of a count of an Augmented Reality (AR) object to be displayed using the display (paragraph 82, At block 408, the device determines an interference state of the user based on the task of the user, and the amount of virtual content displayed in the display of the device), and a ratio of a display area of the AR object to a resolution of the display (paragraph 84, The device 101 computes a total surface area occupied by the AR content displayed in the display 204 and determines a ratio or percentage of display surface occupied by the Virtual or AR content). However, Kazansky et al. do not expressly disclose based on the at least one of the count of AR object and the ratio of the display area of the AR object being greater than or equal to a specified first threshold, determine to display the AR object as an outline using the display; based on the AR object being determined to be displayed as the outline, determine at least one first representative pixel value for the AR object; determine a second representative pixel value for presenting a first region of the image corresponding to a display position of the AR object using the display; based on a difference between the at least one first representative pixel value and the second representative pixel value not being included in a specified threshold range, display the AR object as the outline in accordance with a first display attribute using the at least one first representative pixel value using the display; and based on the difference between the at least one first representative pixel value and the second representative pixel value being included in the specified threshold range, display the AR object as the outline in accordance with a second display attribute different from the first display attribute using the display.
Kazansky et al. disclose identifying at least one of a count of an Augmented Reality (AR) object to be displayed using the display (paragraph 82, At block 408, the device determines an interference state of the user based on the task of the user, and the amount of virtual content displayed in the display of the device) and a ratio of a display area of the AR object to a resolution of the display (paragraph 84, The device 101 computes a total surface area occupied by the AR content displayed in the display 204 and determines a ratio or percentage of display surface occupied by the Virtual or AR content).
Brown et al. disclose determining to display the AR object as an outline using the display; based on the AR object being determined to be displayed as the outline, determining at least one first representative pixel value for the AR object; determining a second representative pixel value for presenting a first region of the image corresponding to a display position of the AR object using the display (paragraph 86, The scooter icon 610 on the other hand, has had its visual properties altered so as to provide less visual obstruction. In particular, the solid fill of the lines in the scooter icon 610 has been removed and replaced with no fill or a transparent fill). The scooter icon comprises pixel values including a first region of the image icon.
Kazansky et al. do not teach based on the at least one of the count of AR object and the ratio of the display area of the AR object being greater than or equal to a specified first threshold, determine to display the AR object as an outline using the display.
Brown et al. is a similar or analogous system to the claimed invention as evidenced Brown et al. teach an augmented reality system for displaying virtual content wherein the design incentive of avoiding distractions while using an AR device would have prompted a predictable variation of Kazansky et al. by applying Brown et al.’s known principle of displaying the AR object as an outline using the display.
In view of the design incentives to avoid distractions while using an AR device, this would have implemented the claimed variation of the prior art system of Kazansky et al.
Therefore, the claimed subject matter would have been obvious to a person having ordinary skill in the art at the time the invention was made.
Osborn et al., who also deal with augmented reality, disclose a method for based on a difference between the at least one first representative pixel value and the second representative pixel value not being included in a specified threshold range, display the AR object as the outline in accordance with a first display attribute using the at least one first representative pixel value using the display (paragraph 21, if the hues of adjacent pixels differ from each other by greater than a threshold variance then it may be determined that the pixels are not included in the same surface, paragraph 28, At 216 the method includes performing a color fill operation on the first identified surface in response to receiving the color fill request, paragraph 39, the method advances to 236 that includes performing a color fill operation on the second identified surface in response to receiving the color fill request); and
based on the difference between the at least one first representative pixel value and the second representative pixel value being included in the specified threshold range, display the AR object as the outline in accordance with a second display attribute different from the first display attribute using the display (paragraph 21, Conversely, if the hues of adjacent pixels do not differ from each other by greater than a threshold variance then it may be determined that the pixels are included in the same surface, paragraph 28, At 216 the method includes performing a color fill operation on the first identified surface in response to receiving the color fill request).
Kazansky et al., Brown et al., and Osborn et al. are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method wherein based on a difference between the at least one first representative pixel value and the second representative pixel value not being included in a specified threshold range, display the AR object as the outline in accordance with a first display attribute using the at least one first representative pixel value using the display; and based on the difference between the at least one first representative pixel value and the second representative pixel value being included in the specified threshold range, display the AR object as the outline in accordance with a second display attribute different from the first display attribute using the display, as taught by Osborn et al., to the Kazansky et al. as modified by Brown et al. system, because a user can fill surfaces in their surrounding environment with a virtual color to adjust the augmented reality environment viewed through the see-through holographic display according to their desire (paragraph 28 of Osborn et al.).
With respect to claim 9, Kazansky et al. as modified by Brown et al. and Osborn et al. disclose a method of displaying an AR object of an electronic device, as executed by the system of claim 1; see rationale for rejection of claim 1.
With respect to claim 16, Kazansky et al. as modified by Brown et al. and Osborn et al. disclose one or more non-transitory computer-readable storage media storing one or more computer programs including computer-executable instructions (Kazansky et al.: paragraph 106, The disk drive unit 1316 includes a computer-readable medium 1322 on which is stored one or more sets of data structures and instructions 1324 (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein) that, when executed by one or more processors of an electronic device individually or collectively, cause the electronic device to perform operations of claim 1; see rationale for rejection of claim 1.
Claim(s) 2, 5-7, 10, 13-14, 17, and 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kazansky et al. (U.S. PGPUB 20170364153) in view of Brown et al. (U.S. PGPUB 20230306690), Osborn et al. (U.S. PGPUB 20170221276), and further in view of Mullins (U.S. PGPUB 20140267407).
With respect to claim 2, Kazansky et al. as modified by Brown et al. and Osborn disclose the electronic device of claim 1. However, Kazansky et al. as modified by Brown et al. and Osborn et al. do not expressly disclose the instructions cause, when being executed by at least one processor individually and/or collectively, the electronic device to: divide the AR object into at least one section defined depending on the outline; and determine the first representative pixel value for each of the at least one section.
Mullins, who also deals with augmented reality, disclose a method wherein the instructions cause, when being executed by at least one processor individually and/or collectively, the electronic device to: divide the AR object into at least one section defined depending on the outline (paragraph 80, The delivery segmentation 202 of the server 110 divides a virtual object model 908 associated with the picture 906 into several segments: segment A, B, and C); and determine the first representative pixel value for each of the at least one section (paragraph 80, segment A of the virtual object is first delivered at 916 so that the device 101 may start rendering the portion corresponding to segment A. In another example, analytics data may indicate that most user first examine the top of the building which corresponds to segment A. Segment B is then delivered and rendered on the device). The virtual object is divided into sections depending on the outline of the building. By delivering and rendering the different portions, this determines pixel values for said portions.
Kazansky et al., Brown et al., Osborn et al., and Mullins are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method wherein the instructions cause, when being executed by at least one processor individually and/or collectively, the electronic device to: divide the AR object into at least one section defined depending on the outline; and determine the first representative pixel value for each of the at least one section, as taught by Mullins, to the Kazansky et al. as modified by Brown et al. and Osborn et al. system, because segments are rendered on the device in the order they are received (paragraph 80 of Mullins), this implementing an efficient way to render the virtual object.
With respect to claim 5, Kazansky et al. as modified by Brown et al., Osborn et al., and Mullins disclose the electronic device of claim 2, wherein the at least one section includes a first section and a second section (Mullins: paragraph 80, analytics data may indicate that most user first examine the top of the building which corresponds to segment A, two different surfaces correspond to the first and second section; see rationale for rejection of claim 2), and wherein the instructions cause, when being executed by at least one processor individually and/or collectively, the electronic device to:
determine whether a difference between the first representative pixel value determined for the first section and the second representative pixel value for the first region of the image is included in the specified threshold range (Osborn et al.: paragraph 20, identifying the first surface may include determining a variance in one or more of the color parameters across a region in the captured image and the surface may be identified based on a variance threshold); and
determine whether the difference between the first representative pixel value determined for the second section and the second representative pixel value for the first region of the image is included in the specified threshold range (Osborn et al.: paragraph 20, identifying the first surface may include determining a variance in one or more of the color parameters across a region in the captured image and the surface may be identified based on a variance threshold, as applied to a second surface); see rationale for rejection of claim 1.
With respect to claim 6, Kazansky et al. as modified by Brown et al., Osborn et al., and Mullins disclose the electronic device of claim 5, wherein the instructions cause, when being executed by at least one processor individually and/or collectively, the electronic device to: based on the difference between the first representative pixel value determined for the first section and the second representative pixel value for the first region of the image not being included in the specified threshold range, display a first outline defining the first section in accordance with the first display attribute (Osborn et al.: paragraph 21, if the hues of adjacent pixels differ from each other by greater than a threshold variance then it may be determined that the pixels are not included in the same surface, paragraph 28, At 216 the method includes performing a color fill operation on the first identified surface in response to receiving the color fill request, paragraph 39, the method advances to 236 that includes performing a color fill operation on the second identified surface in response to receiving the color fill request); and
based on the first representative pixel value determined for the second section and the second representative pixel value for the first region of the image being included in the specified threshold range, display a second outline defining the second section in accordance with the second display attribute (Osborn et al.: paragraph 21, Conversely, if the hues of adjacent pixels do not differ from each other by greater than a threshold variance then it may be determined that the pixels are included in the same surface, paragraph 28, At 216 the method includes performing a color fill operation on the first identified surface in response to receiving the color fill request); see rationale for rejection of claim 1.
With respect to claim 7, Kazansky et al. as modified by Brown et al., Osborn et al., and Mullins disclose the electronic device of claim 6, wherein the instructions cause, when being executed by at least one processor individually and/or collectively, the electronic device to: display the second outline defining the second section in accordance with the second display attribute using a color having a complementary color relationship with the first representative pixel value determined for the second section (Osborn et al.: Fig. 6-8, second outline defining different surfaces with different hatching, paragraph 26, At 212 the method includes altering the colors of the plurality of color swatches in the color palette hologram. In this way, a first color palette may be swapped out with a more desirable second color palette. For instance, a user may wish to quickly flip through several predetermined colors palettes prior to selecting a fill color, desirable second color palette as a complementary color); display the second outline defining the second section in accordance with the second display attribute using a thicker thickness than the first outline displayed in accordance with the first display attribute; or display the second outline defining the second section in accordance with the second display attribute using a higher brightness or higher saturation than the first outline displayed in accordance with the first display attribute. Osborn et al. disclose the first condition, as applied to the surface comprising the second outline.
With respect to claim 10, Kazansky et al. as modified by Brown et al., Osborn et al., and Mullins disclose the method of claim 9, as executed by the system of claim 2; see rationale for rejection of claim 2.
With respect to claim 13, Kazansky et al. as modified by Brown et al., Osborn et al., and Mullins disclose the method of claim 10, as executed by the system of claims 5-6; see rationale for rejection of claims 5-6.
With respect to claim 14, Kazansky et al. as modified by Brown et al., Osborn et al., and Mullins disclose the method of claim 13, as executed by the system of claim 7; see rationale for rejection of claim 7.
With respect to claim 17, Kazansky et al. as modified by Brown et al., Osborn et al., and Mullins disclose the one or more non-transitory computer-readable storage media of claim 16, for implementing the system of claim 2; see rationale for rejection of claim 2.
With respect to claim 19, Kazansky et al. as modified by Brown et al., Osborn et al., and Mullins disclose the one or more non-transitory computer-readable storage media of claim 10, for implementing the system of claims 5-6; see rationale for rejection of claims 5-6.
With respect to claim 20, Kazansky et al. as modified by Brown et al., Osborn et al., and Mullins disclose the one or more non-transitory computer-readable storage media of claim 19, for implementing the system of claim 7; see rationale for rejection of claim 7.
Claim(s) 3-4, 11-12, and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kazansky et al. (U.S. PGPUB 20170364153) in view of Brown et al. (U.S. PGPUB 20230306690), Osborn et al. (U.S. PGPUB 20170221276), Mullins (U.S. PGPUB 20140267407), and further in view of Zhao et al. (U.S. PGPUB 20210337189).
With respect to claim 3, Kazansky et al. as modified by Brown et al., Osborn et al., and Mullins disclose the electronic device of claim 2. However, Kazansky et al. as modified by Brown et al., Osborn et al., and Mullins do not expressly disclose the instructions cause, when being executed by at least one processor individually and/or collectively, the electronic device to: based on a representative value or average value of RGB values of the display for presenting each of the at least one section using the display, determine the first representative pixel value for each of the at least one section; and based on a YCbCr value obtained converted from the representative value or average value of the RGB values, determine the first representative pixel value for each of the at least one section.
Zhao et al., who also deal with augmented reality, disclose a method wherein the instructions cause, when being executed by at least one processor individually and/or collectively, the electronic device to: based on a representative value or average value of RGB values of the display for presenting each of the at least one section using the display, determine the first representative pixel value for each of the at least one section (paragraph 99, the picture may be represented as or include three sample arrays. In an RBG format or color space, the picture includes corresponding red, green, and blue sample arrays); and based on a YCbCr value obtained converted from the representative value or average value of the RGB values, determine the first representative pixel value for each of the at least one section (paragraph 99, A picture in an RGB format may be transformed or converted into a picture in a YCbCr format and vice versa). Zhao et al. disclose determining a first pixel value by representing the augmented reality image using pixels (paragraph 99, A picture is or may be considered as a two-dimensional array or matrix of samples with luminance values. The array is used as an example. A sample in the array may also be referred to as a pixel or a picture element).
Kazansky et al., Brown et al., Osborn et al., Mullins, and Zhao et al. are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method wherein the instructions cause, when being executed by at least one processor individually and/or collectively, the electronic device to: based on a representative value or average value of RGB values of the display for presenting each of the at least one section using the display, determine the first representative pixel value for each of the at least one section; and based on a YCbCr value obtained converted from the representative value or average value of the RGB values, determine the first representative pixel value for each of the at least one section, as taught by Zhao et al., to the Kazansky et al. as modified by Brown et al., Osborn et al., and Mullins system, because for color representation, three color components are usually used (paragraph 99 of Zhao et al.), thus implement color representation using a standard format.
With respect to claim 4, Kazansky et al. as modified by Brown et al., Osborn et al., Mullins, and Zhao et al. disclose the electronic device of claim 1, wherein the instructions cause, when being executed by at least one processor individually and/or collectively, the electronic device to: based on a representative value or average value of RGB values of the display for presenting the first region of the image using the display, determine the second representative pixel value for the first region of the image (Zhao et al.: paragraph 99, the picture may be represented as or include three sample arrays. In an RBG format or color space, the picture includes corresponding red, green, and blue sample arrays); or based on a YCbCr value converted from the representative value or average value of the RGB values, determine the second representative pixel value for the first region of the image (Zhao et al.: paragraph 99, in video coding, each pixel is usually represented in a luma/chroma format or color space, for example, YCbCr, which includes a luminance (luma for short) component indicated by Y (sometimes indicated by L instead) and two chrominance (chroma for short) components indicated by Cb and Cr). Zhao et al. disclose determining a second pixel value by representing the augmented reality image using pixels (paragraph 99, A picture is or may be considered as a two-dimensional array or matrix of samples with luminance values. The array is used as an example. A sample in the array may also be referred to as a pixel or a picture element); see rationale for rejection of claim 3.
With respect to claim 11, Kazansky et al. as modified by Brown et al., Osborn et al., Mullins, and Zhao et al. disclose the method of claim 10, as executed by the system of claim 3; see rationale for rejection of claim 3.
With respect to claim 12, Kazansky et al. as modified by Brown et al., Osborn et al., Mullins, and Zhao et al. disclose the method of claim 9, as executed by the system of claim 4; see rationale for rejection of claim 4.
With respect to claim 18, Kazansky et al. as modified by Brown et al., Osborn et al., Mullins, and Zhao et al. disclose the one or more non-transitory computer-readable storage media of claim 17, for implementing the system of claim 3; see rationale for rejection of claim 3.
Claim(s) 8 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kazansky et al. (U.S. PGPUB 20170364153) in view of Brown et al. (U.S. PGPUB 20230306690), Osborn et al. (U.S. PGPUB 20170221276), and further in view of Xie (U.S. PGPUB 20230398453).
With respect to claim 8, Kazansky et al. as modified by Brown et al. and Osborn et al. disclose the electronic device of claim 1. However, Kazansky et al. as modified by Brown et al. and Osborn et al. do not expressly disclose the instructions cause, when being executed by at least one processor individually and/or collectively, the electronic device to: determine whether the at least one of the count of the AR object and the ratio of the display area of the AR object are greater than or equal to a specified second threshold greater than the specified first threshold; based on the at least one of the count of the AR object and the ratio of the display area of the AR object being greater than or equal to the specified second threshold, determine a third representative pixel value for the entire region of the image; and display the AR object as the outline in accordance with a third display attribute using a color having a complementary relationship with the third representative pixel value.
Xie, who also deals with augmented reality, discloses a method wherein the instructions cause, when being executed by at least one processor individually and/or collectively, the electronic device to: determine whether the at least one of the count of the AR object and the ratio of the display area of the AR object are greater than or equal to a specified second threshold greater than the specified first threshold (paragraph 132, the following processing may alternatively be performed for any type of virtual items in the first virtual item list: when the remaining quantity of virtual items of the any type is not zero, applying a color mode to the any type of virtual items (namely, displaying virtual items in color); and when the remaining quantity of virtual items of the any type decreases to zero); based on the at least one of the count of the AR object and the ratio of the display area of the AR object being greater than or equal to the specified second threshold, determine a third representative pixel value for the entire region of the image (paragraph 132, applying a color mode to the any type of virtual items); and display the AR object as the outline in accordance with a third display attribute using a color having a complementary relationship with the third representative pixel value (paragraph 132, namely, displaying virtual items in color). Xie discloses when the count of an AR object is greater than a second threshold, i.e., not zero, which is greater than a first threshold equal to zero.
Kazansky et al., Brown et al., Osborn et al., and Xie are in the same field of endeavor, namely computer graphics.
Before the effective filing date of the claimed invention, it would have been obvious to apply the method wherein the instructions cause, when being executed by at least one processor individually and/or collectively, the electronic device to: determine whether the at least one of the count of the AR object and the ratio of the display area of the AR object are greater than or equal to a specified second threshold greater than the specified first threshold; based on the at least one of the count of the AR object and the ratio of the display area of the AR object being greater than or equal to the specified second threshold, determine a third representative pixel value for the entire region of the image; and display the AR object as the outline in accordance with a third display attribute using a color having a complementary relationship with the third representative pixel value, as taught by Xie, when applied to the Kazansky et al. as modified by Brown et al. and Osborn et al. system, comprising a complementary color relationship (Osborn et al.: paragraph 26, At 212 the method includes altering the colors of the plurality of color swatches in the color palette hologram. In this way, a first color palette may be swapped out with a more desirable second color palette. For instance, a user may wish to quickly flip through several predetermined colors palettes prior to selecting a fill color, desirable second color palette as a complementary color) because different operation modes are set to discard and use virtual items, thereby improving human-computer interaction efficiency (paragraph 134 of Xie).
With respect to claim 15, Kazansky et al. as modified by Brown et al., Osborn et al., and Xie disclose the method of claim 9, as executed by the system of claim 8; see rationale for rejection of claim 8.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW GUS YANG whose telephone number is (571)272-5514. The examiner can normally be reached M-F 9 AM - 5:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW G YANG/Primary Examiner, Art Unit 2614
8/29/26
Prosecution Insights