DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-19 of U.S. Patent No. US 12190457 B2 in view of Cho (US 20190213416 A1).
Application 18959194
Claim 1
US 12190457 B2
Claim 16
1. An augmented reality (AR) wearable device comprising:
16. An augmented reality (AR) wearable device comprising:
at least one processor; and
one or more processors; and
one or more memories storing instructions that, when executed by the at least one processor, configure the AR wearable device to perform operations comprising:
one or more memories storing instructions that, when executed by the one or more processors, configure the AR wearable device to perform operations comprising:
capturing, by an image capturing device of the AR wearable device, an image corresponding to a user view of a real-world scene;
capturing, by an image capturing device of the AR wearable device, an image corresponding to a first user view of a real-world scene;
sending the image to a backend computer via a wireless communication link;
receiving indications of a plurality of objects within the image, indications of the locations of the plurality of objects, and indications of labels for the plurality of objects;
processing the image to determine locations of a plurality of objects within the image, and labels for the plurality of objects;
selecting a group of objects of the plurality of objects based at least on the labels for the plurality of objects indicating a same label;
selecting a group of objects of the plurality of objects based at least on the labels of each object of the group of objects indicating a same label;
counting a number of objects in the group of objects to generate a count for the group of objects;
counting a number of objects in the group of objects to generate a count for the group of objects;
identifying an area of the image where additional objects are obscured, wherein the additional objects are excluded from the count of the group of objects; and
identifying, by processing the image, an area of the image where additional objects are obscured by the group of objects, wherein the additional objects are excluded from the count of the group of objects; and
displaying, on a display of the AR wearable device, indications that the number of objects in the group of objects have been counted, an indication that the additional objects are obscured, and an indication of the count.
displaying, on a display of the AR wearable device, indications that the objects in the group of objects have been counted, an indication that the additional objects are obscured, and an indication of the count, wherein locations of the indications on the display are based on the locations of the objects in the group of objects and a second user view of the real-world scene.
2-20
1-15 and 17-19
The claim 16 of U.S. Patent No. US 12190457 B2 fails to explicitly disclose “processing the image to determine locations of a plurality of objects within the image, and labels for the plurality of objects”. In same field of endeavor, Cho (US 20190213416 A1) teaches “processing the image to determine locations of a plurality of objects within the image, and labels for the plurality of objects” in Fig. 4, Fig. 6, and paragraphs [0069], [0096], and [0113]. The motivation for doing so would have been to extract at least one feature from an image obtained by the camera module 180 to identify what is a main subject for photography in the image; to identify objects associated with food from the image; divide the objects from the image as taught by Cho in Fig. 4, and paragraphs [0069] and[0096].
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-10, 12-13, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Cho (US 20190213416 A1) and in view of Trott (US 20190130206 A1).
Regarding to claim 1, Cho discloses an augmented reality (AR) wearable device (Fig. 1; [0038]: the electronic device 101 in the network environment 100 communicates with an electronic device 102, an electronic device 104, and a server 108 via a wireless communication network; [0058]: the electronic device includes a wearable device; [0076]: a wearable device; [0119]: a visual effect is associated with Augmented Reality , i.e. AR; the electronic device 101 displays an indication for the visual effect associated with AR; [0160]: a wearable device, such as smart glasses) comprising:
at least one processor (Fig. 1; [0038]: a processor; [0039]: processor; CPU); and
one or more memories storing instructions that, when executed by the at least one processor, configure the AR wearable device to perform operations (Fig. 1; [0038]: a processor and memory 130; [0039]: processor; CPU; Fig. 1; [0041]: the memory 130 may store various data used by at least one component; [0061]: a processor of the machine invokes at least one of the one or more instructions stored in the storage medium, and execute it) comprising:
capturing, by an image capturing device of the AR wearable device, an image corresponding to a user view of a real-world scene ([0051]: the camera module 180 captures a still image or moving images of a real world scene; [0058]: the electronic devices include a camera; Fig. 4; [0094]: the images are obtained by at least one camera included in the camera module 180; [0095]: the at least one light is emitted in a direction corresponding to the view, e.g., Field Of View FOV of the at least one camera; [0145]: the captured image; the image is displayed in the user interface 1440 after capture; [0160]: a wearable device, such as smart glasses);
processing the image to determine locations of a plurality of objects within the image, and labels for the plurality of objects ([0069]: extract at least one feature from an image obtained by the camera module 180 to identify what is a main subject for photography in the image; Fig. 4; [0096]: the processor 120 identifies objects associated with food from the image; the processor 120 extracts at least one feature from the image; obtain, through image processing, information on a boundary of each of a plurality of objects in the image; Fig. 6; [0113]: another indication 630 provides further information on the recommended intake in the partial image 600; the indication 630 is configured with a combination of numbers or letters in order;
PNG
media_image1.png
300
400
media_image1.png
Greyscale
);
selecting a group of objects of the plurality of objects based at least on the labels (Fig. 8; [0116]: the processor 120 displays each of the indications along with each of the objects; the image 800 includes an object 810 indicating a hamburger, an object 820 indicating a steak, an object 830 indicating French fries, an object 840 indicating bread, an object 850 indicating a cafe latte, and an object 860 indicating a ricotta cheese salad; the object 860 is not selected and is not associated with an indication; thereby preventing the user from eating food to which the user is allergic; [0117]: the processor 120 blurs an object which does not match a user's preference; Fig. 19; [0168]:
PNG
media_image2.png
106
229
media_image2.png
Greyscale
) for the plurality of objects indicating a same label (Fig. 8; [0116]: each piece of a group of French fries is an object and indicates a same label of 830 and dot lines as illustrated in Fig. 8;
PNG
media_image3.png
122
117
media_image3.png
Greyscale
; portion of French fries is an object and indicate the same label; Fig. 12; [0135]: display an indication 1225 indicating the recommended intake of the ricotta cheese salad within an object 860 associated with the ricotta cheese salad; each piece of salad is an object and indicates a same label 1225 as illustrated in Fig. 12); and
displaying, on a display of the AR wearable device, indications that the number of objects in the group of objects have been counted (Fig. 8; [0118]: display objects and indications;
PNG
media_image4.png
437
609
media_image4.png
Greyscale
; display at least one of the objects or the indications; display visual effects such as the texture, the color, or the brightness of each of the objects associated with each of the food to different levels, depending on nutrients of each of the food; Fig. 8; [0119]: the electronic device 101 shows a visual effect associated with Augmented Reality (AR); Fig. 17; [0159]: display an image 1700 including the objects associated with the food;
PNG
media_image5.png
643
642
media_image5.png
Greyscale
).
Cho fails to explicitly disclose:
counting a number of objects in the group of objects to generate a count for the group of objects;
identifying an area of the image where additional objects are obscured, wherein the additional objects are excluded from the count of the group of objects;
an indication that the additional objects are obscured, and an indication of the count.
In same field of endeavor, Trott teaches:
counting a number of objects in the group of objects to generate a count for the group of objects (Fig. 1; [0022]: count the objects from the image; [0027]: counting module 330 counts the objects in an image that match criteria, including labels; counting module 330 generates a count 360 of the objects from image 340 matching a criteria; counting module 330 generates a set of object outlines 370 including an outline of each of the counted objects; [0033]: counter 460 generates a count 470 and a set of outlines 480 of the objects; Fig. 2; [0061]: the object outlines are output by overlaying the object outlines on the image);
identifying an area of the image where additional objects are obscured (Fig. 11; [0067]: ground truth values are 4 people; Trott teaches two people are obscured in SoftCount, upDown, and IRLC as illustrated in Fig. 11;
PNG
media_image6.png
222
693
media_image6.png
Greyscale
; each of the results for the UpDown counter is shown with its count value above the image and the opacity of the various outlines corresponds to the attention focus generated by the UpDown counter), wherein the additional objects are excluded from the count of the group of objects (Fig. 11; [0067]: for UpDown, one obscured motorcycle is not counted;
PNG
media_image7.png
130
500
media_image7.png
Greyscale
);
an indication that the additional objects are obscured, and an indication of the count (Fig. 11; [0067]: each of the results for IRLC 700 is shown with its count value above the image and the outlines indicate the counted objects; two people are obscured in SoftCount, upDown, and IRLC as illustrated in Fig. 11;
PNG
media_image6.png
222
693
media_image6.png
Greyscale
).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Cho to include counting a number of objects in the group of objects to generate a count for the group of objects; identifying an area of the image where additional objects are obscured, wherein the additional objects are excluded from the count of the group of objects; an indication that the additional objects are obscured, and an indication of the count as taught by Trott. The motivation for doing so would have been to count the objects in an image that match criteria; to generate a count 470 and a set of outlines 480 of the objects selected for counting; to demonstrate both the best accuracy and RMSE of each of the three counters as taught by Trott in Fig. 13 and paragraphs [0027], [0033], and [0076].
Regarding to claim 3, Cho in view of Trott discloses the AR wearable device of claim 1, wherein locations of the indications on the display are based on the locations of the objects in the group of objects and a second user view of the real-world scene (Choi; [0095]: the at least one light is emitted in a direction corresponding to the view, e.g., Field Of View FOV of the at least one camera; Fig. 8; [0119]: the electronic device 101 shows a visual effect associated with Augmented Reality (AR); Fig. 17; [0159]: display an image 1700 including the objects associated with the food;
PNG
media_image5.png
643
642
media_image5.png
Greyscale
).
Regarding to claim 4, Cho in view of Trott discloses the AR wearable device of claim 1, wherein the processing further comprises:
sending, via a wireless communication link, an instruction for a computing device to process the image (Cho; Fig. 1; [0038]: the electronic device 101 in the network environment 100 communicates with an electronic device 102, an electronic device 104, and a server 108 via a wireless communication network; [0057]: data are transmitted and received between the electronic device 101 and the external electronic device 104 via the server 108 coupled with the second network 199; the electronic device 101 requests the one or more external electronic devices to perform at least part of the function or the service using client-server computing technology); and
receiving input indicating the locations of the plurality of objects and labels for the plurality of objects (Cho; [0057]: data are transmitted and received between the electronic device 101 and the external electronic device 104 via the server 108 coupled with the second network 199 using client-server computing technology; Fig. 4; [0096]: obtain, through image processing, information on a boundary of each of a plurality of objects in the image; divide the objects from the image in order to recognize each of the objects; [0102]: add the total calories of food A, and the total calories of food B; Fig. 7; [0115]: each of the indications are displayed with a shape of at least a portion of the objects in a position as illustrated in Fig. 7; Fig. 8; [0116]: receive and display each of the indications along with each of the objects; the image 800 includes an object 810 indicating a hamburger, an object 820 indicating a steak, an object 830 indicating French fries, an object 840 indicating bread, an object 850 indicating a cafe latte, and an object 860 indicating a ricotta cheese salad; Fig. 17; [0159]: receive and display an image 1700 including the objects associated with the food;
PNG
media_image5.png
643
642
media_image5.png
Greyscale
).
Regarding to claim 5, Cho in view of Trott discloses the AR wearable device of claim 1 wherein the selecting is further based on at least one of: the group of objects being located in a central portion of the image (at least one of is optional; Cho; Fig. 6; [0113]: a partial image 600 include an indication 620; a visual outline is surrounding an identified food item; display the indications 620 superimposed on an object 610; the indication 620 is in central of the image), the group of objects being located within a threshold distance of one another (at least one of is optional; Cho; Fig. 8; [0117]: the processor 120 blur objects other than the objects 810, 820, 830, 840, 850, and 860 in the image 800;
PNG
media_image8.png
434
561
media_image8.png
Greyscale
), the group of objects being within a threshold distance of a user of the AR wearable device (at least one of is optional; Trott; [0035]: receives a caption; [0036]: multiple captions 530 are provided; [0042]: object selector 760 receives the logit values k from logits module 740 and determines whether another object is to be counted; selects the object with the highest logit value), the group of objects having a same label, or an area selected by the user (at least one of is optional; Cho; Fig. 4; [0100]: the processor 120 selects and identifies the objects associated with the food from the image through an operation, such as operation 430 in FIG. 4.).
Same motivation of claim 1 is applied here.
Regarding to claim 6, Cho in view of Trott discloses the AR wearable device of claim 1 wherein the indications that the number of objects in the group of objects have been counted are boundary boxes around each of the number of objects of the group of objects (Trott; Fig. 1; [0022]: count the objects from the image; [0027]: counting module 330 counts the objects in an image; counting module 330 generates a count 360 of the objects from image 340 matching a criteria and a set of object outlines 370 including an outline of each of the counted objects; [0033]: counter 460 generates a count 470 and a set of outlines 480 of the objects selected for counting; Fig. 2; [0061]: the object outlines are output by overlaying the object outlines on the image received during process 810, such as is shown by bounding boxes 240 in FIG. 2;
PNG
media_image9.png
350
358
media_image9.png
Greyscale
). Same motivation of claim 1 is applied here.
Regarding to claim 7, Cho in view of Trott discloses the AR wearable device of claim 1 wherein probabilities are associated with the labels (Trott; [0044]: E is the baseline count error obtained by selecting the object randomly based on the probability values p.sup.t, with training being performed to reinforce the selection of the randomly selected objects; [0055]: two candidate objects are selected based on Equation 7 and randomly based on the p.sup.t probabilities of Equation 9 with both selected objects being used for separate counting results in parallel), the probabilities indicating a likelihood a corresponding object of the group of objects is correctly labeled (Trott; [0036]: multiple captions 530 are provided; scorer 550 generates a scoring of how well each object in object embeddings v matches each of the embedded captions; [0044]: E is the baseline count error obtained by selecting the object randomly based on the probability values p.sup.t, with training being performed to reinforce the selection of the randomly selected objects; [0051]: the scoring vector includes a score indicating how well each of the objects in the generated object embeddings generated during process 820 matches the criteria encoded in the question embeddings generated during process 840; [0055]: during training, two candidate objects are selected based on Equation 7 and randomly based on the p.sup.t probabilities of Equation 9 with both selected objects being used for separate counting results in parallel), and wherein boundary boxes indicate the probabilities associated with the labels of the group of objects (Trott; [0036]: multiple captions 530 are provided and scorer 550 generates a scoring of how well each object in object embeddings v matches each of the embedded captions; [0044]: E is the baseline count error obtained by selecting the object randomly based on the probability values p.sup.t, with training being performed to reinforce the selection of the randomly selected objects; Fig. 2; [0061]: the object outlines are output by overlaying the object outlines on the image received during process 810, such as is shown by bounding boxes 240 in FIG. 2;
PNG
media_image9.png
350
358
media_image9.png
Greyscale
; [0066]: a scorer determines which how well the caption corresponds to each of the objects determined during process 820).
Regarding to claim 8, Cho in view of Trott discloses the AR wearable device of claim 1, wherein the operations further comprise:
receiving an indication from a user to count objects having a label of the labels (Cho; Fig. 6; [0113]: a partial image 600 includes an indication 620, which takes the form of a visual outline surrounding an identified food item).
Regarding to claim 9, Cho in view of Trott discloses the AR wearable device of claim 1, wherein the operations further comprise:
before the identifying, receiving a selection of an area of the image, and wherein the identifying is within the area of the image (Cho; Fig. 4; [0094]: the processor 120 displays an image obtained by a camera module 180).
Cho in view of Trott further discloses before the identifying, receiving a selection of an area of the image, and wherein the identifying is within the area of the image (Trott; Fig. 1; [0020]: an image is received; [0022]: count the objects from the image matching the question).
Regarding to claim 10, Cho in view of Trott discloses the AR wearable device of claim 1, wherein the labels identify names of the plurality of objects (Cho; [0117]: the processor 120 blurs an object which does not match a user's preference; Fig. 17; [0159]: an object related to a steak and indicates the total calories of a steak; an object related to French fries;
PNG
media_image5.png
643
642
media_image5.png
Greyscale
; Fig. 19; [0168]:
PNG
media_image2.png
106
229
media_image2.png
Greyscale
).
Regarding to claim 12, Cho in view of Trott discloses the AR wearable device of claim 1, wherein the operations further comprise:
determining values for objects of the group of objects (Cho; Fig. 17; [0159]: second information 1720 is associated with an object related to a steak, and indicates the total calories of a steak, i.e. 504 kcal/200 g, second information 1730 is associated with an object related to French fries, and indicates the total calories of French fries, i.e. 229 kcal/100 g, second information 1740 is associated with an object related to a cafe latte, and indicates the total calories of a cafe latte, i.e. 105 kcal/200 ml);
determining a total value for the group of objects based on the values (Cho; Fig. 17; [0159]: The user may recognize the total calories of each of the food associated with the objects included in the image 1700 through the image 1700); and
displaying, on the display, an indication of the total value (Cho; Fig. 17; [0159]: second information 1720 is associated with an object related to a steak, and indicates the total calories of a steak, i.e. 504 kcal/200 g, second information 1730 is associated with an object related to French fries, and indicates the total calories of French fries, i.e. 229 kcal/100 g, second information 1740 is associated with an object related to a cafe latte, and indicates the total calories of a cafe latte, i.e. 105 kcal/200 ml ; Fig. 19; [0168]: the processor 120 displays information 1910 on the user's intake in a user interface 1900 of the application).
Regarding to claim 13, Cho in view of Trott discloses the AR wearable device of claim 12, wherein the values for the objects are currency values and the objects are currency or the values for the objects are calorie estimates and the objects are food (Cho; Fig. 17; [0159]: second information 1720 is associated with an object related to a steak, and indicates the total calories of a steak, i.e. 504 kcal/200 g, second information 1730 is associated with an object related to French fries, and indicates the total calories of French fries, i.e. 229 kcal/100 g, second information 1740 is associated with an object related to a cafe latte, and indicates the total calories of a cafe latte, i.e. 105 kcal/200 ml).
Regarding to claim 17, Cho discloses a method performed by an apparatus of an augmented reality (AR) wearable device (Fig. 1; [0038]: a processor and memory 130; the electronic device 101 in the network environment 100 communicates with an electronic device 102, an electronic device 104, and a server 108 via a wireless communication network; [0039]: processor; CPU; Fig. 1; [0041]: the memory 130 may store various data used by at least one component; [0058]: the electronic device includes a wearable device; [0076]: a wearable device; [0061]: a processor of the machine invokes at least one of the one or more instructions stored in the storage medium, and execute it; [0119]: a visual effect is associated with Augmented Reality , i.e. AR; the electronic device 101 displays an indication for the visual effect associated with AR; [0160]: a wearable device, such as smart glasses) comprising:
The rest claim limitations are similar to claim limitations recited in claim 1. Therefore, same rational used to reject claim 1 is also used to reject claim 17.
Regarding to claim 18, Cho in view of Trott discloses the method of claim 17,
The rest claim limitations are similar to claim limitations recited in claim 5. Therefore, same rational used to reject claim 5 is also used to reject claim 18.
Regarding to claim 19, Cho discloses a non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by at least one processor of an apparatus of an augmented reality (AR) wearable device, cause the at least one processor to perform operations (Fig. 1; [0038]: a processor and memory 130; the electronic device 101 in the network environment 100 communicates with an electronic device 102, an electronic device 104, and a server 108 via a wireless communication network; [0039]: processor; CPU; Fig. 1; [0041]: the memory 130 may store various data used by at least one component; [0058]: the electronic device includes a wearable device; [0076]: a wearable device; [0061]: a processor of the machine invokes at least one of the one or more instructions stored in the storage medium, and execute it; [0119]: a visual effect is associated with Augmented Reality , i.e. AR; the electronic device 101 displays an indication for the visual effect associated with AR; [0160]: a wearable device, such as smart glasses) comprising:
The rest claim limitations are similar to claim limitations recited in claim 1. Therefore, same rational used to reject claim 1 is also used to reject claim 19.
Regarding to claim 20, Cho in view of Trott discloses the non-transitory computer-readable storage medium of claim 19,
The rest claim limitations are similar to claim limitations recited in claim 5. Therefore, same rational used to reject claim 5 is also used to reject claim 20.
Claims 2, 11, and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Cho (US 20190213416 A1) in view of Trott (US 20190130206 A1), and further in view of Chirakansakcharoen (US 20160110902 A1).
Regarding to claim 2, Cho in view of Trott discloses the AR wearable device of claim 1,
Cho in view of Trott fails to explicitly disclose:
wherein the identifying further includes identifying three-dimensional coordinates within a world coordinate system for the plurality of objects.
In same field of endeavor, Chirakansakcharoen teaches:
wherein the identifying further includes identifying three-dimensional coordinates within a world coordinate system for the plurality of objects ([0040]: the entire environment is modeled in 3D; Fig. 7; [0056]: the arrows 735 cause the display area to be adjusted in the corresponding direction; [0057]: navigate across a substantially continuous combined image information, e.g., a 3D modeling of the environment; the UI allows the viewer to select the particular shelving unit, and then navigate along the selected shelving unit.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Cho in view of Trott to include wherein the identifying further includes identifying three-dimensional coordinates within a world coordinate system for the plurality of objects as taught by Chirakansakcharoen. The motivation for doing so would have been to identify items included in an area 255 on the shelf 140A.sub.1; to model the entire environment in 3D; to improve the overall coverage of the combined image information 435; to improve the coverage and/or quality of the combined image information when displayed for other persons as taught by Chirakansakcharoen in paragraphs Fig. 2 and [0024], [0040], [0042], and [0047].
Regarding to claim 11, Cho in view of Trott discloses the AR wearable device of claim 1, wherein the counting the number of objects further (Trott; [0041]: interaction module 750 receives information on each object previously selected by object selector 760; interaction module 750 is able to learn not to count an object twice) comprises:
determining that a subset of objects of the group of objects were previously counted (Trott; [0041]: interaction module 750 receives information on each object previously selected by object selector 760); and
generating a count of objects of the group of objects which were not previously counted (Trott; [0041]: interaction module 750 receives information on each object previously selected by object selector 760; interaction module 750 is able to learn not to count an object twice).
Cho in view of Trott fails to explicitly disclose:
wherein the identifying further includes identifying three-dimensional coordinates within a world coordinate system for the plurality of objects.
In same field of endeavor, Chirakansakcharoen teaches wherein the identifying further includes identifying three-dimensional coordinates within a world coordinate system for the plurality of objects ([0040]: the entire environment is modeled in 3D; Fig. 7; [0056]: the arrows 735 cause the display area to be adjusted in the corresponding direction; [0057]: navigate across a substantially continuous combined image information, e.g., a 3D modeling of the environment; the UI allows the viewer to select the particular shelving unit, and then navigate along the selected shelving unit.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Cho in view of Trott to include wherein the identifying further includes identifying three-dimensional coordinates within a world coordinate system for the plurality of objects as taught by Chirakansakcharoen. The motivation for doing so would have been to identify items included in an area 255 on the shelf 140A.sub.1; to model the entire environment in 3D; to improve the overall coverage of the combined image information 435; to improve the coverage and/or quality of the combined image information when displayed for other persons as taught by Chirakansakcharoen in paragraphs Fig. 2 and [0024], [0040], [0042], and [0047].
Regarding to claim 15, Cho in view of Trott discloses the AR wearable device of claim 1, wherein the image is a first image, the count is a first count, the group of objects is a first group of objects, the plurality of objects is a first plurality of objects, the labels are first labels, the indications are first indications (Trott; [0041]: interaction module 750 receives information on each object previously selected by object selector 760; interaction module 750 is able to learn not to count an object twice), and wherein the operations further comprise:
capturing, by the image capturing device, a second image corresponding to a third user view of the real-world scene (Cho; [0051]: the camera module 180 captures a still image or moving images of a real world scene; [0058]: the electronic devices include a camera; Fig. 4; [0094]: the images are obtained by at least one camera included in the camera module 180; [0095]: the at least one light is emitted in a direction corresponding to the view, e.g., Field Of View FOV of the at least one camera; [0145]: the captured image; the image is displayed in the user interface 1440 after capture; [0160]: a wearable device, such as smart glasses);
processing the second image to determine indications of a second plurality of objects within the second image (Cho; [0069]: extract at least one feature from second image obtained by the camera module 180 to identify what is a main subject for photography in the image; Fig. 4; [0096]: the processor 120 identifies objects associated with food from the second image; the processor 120 extracts at least one feature from the second image; obtain, through image processing, information on a boundary of each of a plurality of objects in the image; Fig. 6; [0113]: another indication 630 provides further information on the recommended intake in the partial image 600; the indication 630 is configured with a combination of numbers or letters in order;
PNG
media_image1.png
300
400
media_image1.png
Greyscale
), second three-dimensional coordinates within the world coordinate system for the second plurality of objects, and second labels for the second plurality of objects (Cho; [0057]: data are transmitted and received between the electronic device 101 and the external electronic device 104 via the server 108 coupled with the second network 199 using client-server computing technology; Fig. 4; [0096]: obtain, through image processing, information on a boundary of each of a plurality of objects in the image; divide the objects from the image in order to recognize each of the objects; [0102]: add the total calories of food A, and the total calories of food B; Fig. 7; [0115]: each of the indications are displayed with a shape of at least a portion of the objects in a position as illustrated in Fig. 7; Fig. 8; [0116]: receive and display each of the indications along with each of the objects; the image 800 includes an object 810 indicating a hamburger, an object 820 indicating a steak, an object 830 indicating French fries, an object 840 indicating bread, an object 850 indicating a cafe latte, and an object 860 indicating a ricotta cheese salad; Fig. 17; [0159]: receive and display an image 1700 including the objects associated with the food;
PNG
media_image5.png
643
642
media_image5.png
Greyscale
);
selecting a second group of objects of the second plurality of objects based on objects of the second group of objects being in a same group of objects as the first group of objects ( Cho; Fig. 8; [0116]: the processor 120 displays each of the indications along with each of the objects; the image 800 includes an object 810 indicating a hamburger, an object 820 indicating a steak, an object 830 indicating French fries, an object 840 indicating bread, an object 850 indicating a cafe latte, and an object 860 indicating a ricotta cheese salad; the object 860 is not selected and is not associated with an indication; thereby preventing the user from eating food to which the user is allergic; [0117]: the processor 120 blurs an object which does not match a user's preference; Fig. 19; [0168]:
PNG
media_image2.png
106
229
media_image2.png
Greyscale
);
counting objects of the second group of objects which were not previously counted in the first group of objects to generate a second count (Trott; Fig. 1; [0022]: count the objects from the image; [0027]: counting module 330 counts the objects in an image that match criteria, including labels; counting module 330 generates a count 360 of the objects from image 340 matching a criteria; counting module 330 generates a set of object outlines 370 including an outline of each of the counted objects; [0033]: counter 460 generates a count 470 and a set of outlines 480 of the objects; Fig. 2; [0061]: the object outlines are output by overlaying the object outlines on the image); and
displaying, on the display of the AR wearable device, second indications that the second group of objects which were not previously counted have been counted and an indication of the second count (Trott; [0041]: interaction module 750 receives information on each object previously selected by object selector 760; interaction module 750 is able to learn not to count an object twice),
Cho in view of Trott fails to explicitly disclose:
wherein the processing further comprises processing the image to determine first three-dimensional coordinates within a world coordinate system for the first plurality of objects,
wherein locations of the second indications are based on the second three-dimensional coordinates of the second group of objects and a third user view of the real-world scene.
In same field of endeavor, Chirakansakcharoen teaches:
wherein the processing further comprises processing the image to determine first three-dimensional coordinates within a world coordinate system for the first plurality of objects ([0040]: the entire environment is modeled in 3D; Fig. 7; [0056]: the arrows 735 cause the display area to be adjusted in the corresponding direction; [0057]: navigate across a substantially continuous combined image information, e.g., a 3D modeling of the environment; the UI may allow the viewer to select the particular shelving unit, and then navigate along the selected shelving unit.),
wherein locations of the second indications are based on the second three-dimensional coordinates of the second group of objects and a third user view of the real-world scene ([0040]: the entire environment is modeled in 3D; Fig. 7; [0056]: the arrows 735 cause the display area to be adjusted in the corresponding direction; [0057]: navigate across a substantially continuous combined image information, e.g., a 3D modeling of the environment; the UI allows the viewer to select the particular shelving unit or aisle, and then navigate along the selected shelving unit.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Cho in view of Trott to include wherein the processing further comprises processing the image to determine first three-dimensional coordinates within a world coordinate system for the first plurality of objects,
wherein locations of the second indications are based on the second three-dimensional coordinates of the second group of objects and a third user view of the real-world scene as taught by Chirakansakcharoen. The motivation for doing so would have been to identify items included in an area 255 on the shelf 140A.sub.1; to model the entire environment in 3D; to improve the overall coverage of the combined image information 435; to improve the coverage and/or quality of the combined image information when displayed for other persons as taught by Chirakansakcharoen in paragraphs Fig. 2 and [0024], [0040], [0042], and [0047].
Regarding to claim 16, Cho in view of Trott and Chirakansakcharoen discloses the AR wearable device of claim 15, further comprising:
displaying, on the display of the AR wearable device, third indications that the second group of objects which were previously counted have not been counted in the second count and an indication of the second count plus the first count (Cho; Fig. 17; Fig. 18; [0159]: second information 1730 is associated with an object related to French fries and indicates the total calories of French fries;
PNG
media_image10.png
480
531
media_image10.png
Greyscale
), wherein locations of the third indications are based on the second three-dimensional coordinates of the second group of objects which were previously counted and the third user view of the real-world scene (Chirakansakcharoen; [0040]: the entire environment may be modeled in 3D; Fig. 7; [0056]: the arrows 735 cause the display area to be adjusted in the corresponding direction; [0057]: navigate across a substantially continuous combined image information (e.g., a 3D modeling of the environment); the UI may allow the viewer to select the particular shelving unit (or aisle), and then navigate along the selected shelving unit).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Cho (US 20190213416 A1) in view of Trott (US 20190130206 A1), and further in view of Kimura (US 20220036093 A1).
Regarding to claim 14, Cho in view of Trott discloses the AR wearable device of claim 1, wherein the number of objects are people, and wherein the operations (Trott; Fig. 10; [0060]: three people are in the image as illustrated in Fig. 10) further comprise:
Cho in view of Trott fails to explicitly disclose:
determining, by processing the image, a venue where the people are located; and
in response to the count exceeding or transgressing a threshold of a number of people for the venue, displaying, on the display, an indication that the number of people has exceeded a capacity for the venue.
In same field of endeavor, Kimura teaches:
determining, by processing the image, a venue where the people are located (Fig. 5; [0040]: generates a figure 505 corresponding to a graph based on the number of objects that passes through a single passage line 501 in an IN direction 502 for each of the five location “location 1” to “location 5” on the passage line 501); and
in response to the count exceeding or transgressing a threshold of a number of people for the venue, displaying, on the display, an indication that the number of people has exceeded a capacity for the venue (Fig. 5; [0040]: counting the number of objects that have passed through the single passage line 501 in the OUT direction 503 during the counting period; Fig. 6A; Fig. 6B; [0041]: a graph 600a illustrated in FIG. 6A is a graph indicating a count result of the objects that have passed through the passage line 501 in the IN direction 502 illustrated in FIG. 5 for each of the five locations “location 1” to “location 5” in FIG. 5. In addition, a graph 600b illustrated in FIG. 6B is a graph indicating a count result of the objects that have passed through the passage line 501 in the OUT direction 503 illustrated in FIG. 5 for each of the five locations “location 1” to “location 5” in FIG. 5.; [0042]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Cho in view of Trott to include determining, by processing the image, a venue where the people are located; and in response to the count exceeding or transgressing a threshold of a number of people for the venue, displaying, on the display, an indication that the number of people has exceeded a capacity for the venue as taught by Kimura. The motivation for doing so would have been to count the number of objects that have passed through the single passage line 501 in the OUT direction 503 during the counting period as taught by Kimura in Fig. 5 and paragraphs [0040-0042].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Hai Tao Sun whose telephone number is (571)272-5630. The examiner can normally be reached 9:00AM-6:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 5712727642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAI TAO SUN/Primary Examiner, Art Unit 2616
Prosecution Insights