DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicants’ submission filed on 20 July 2026 has been entered. Claims 1-3, 8, 10, 12, 13, 15 and 16 have been amended. Claims 17-20 have been newly added. Claims 1-20 are currently pending.
Response to Arguments
Applicants’ arguments filed 20 July 2026 have been fully and carefully considered but they are not persuasive. Applicant argues on pages 1-2 of the Remarks that the combination of references fails to disclose the limitation, “acquire characteristic information related to an angle and a distance at which the target object can be gazed at” as recited in the independent claims.
The Examiner respectfully disagrees.
Yonaha discloses in paragraph [0067], “the imaging range information storage unit 240 stores imaging range information that specifies the imaging range that each of the multiple imaging devices 100 will capture. As a result, the target area identification unit 224 can exclude areas of space in the direction of the person's line of sight from the line of sight area 330 where no objects exist, and can identify the image area in which the object seen by the person was captured” and in paragraph [0069], “Based on the imaging range information described in relation to Figure 4, the target area identification unit 224 identifies the line-of-sight area 332, which is the image area in which the object seen by the person is captured, as the target area by excluding the area of space in which no object exists.”
Yonaha further discloses in paragraph [0070], “the target area identification unit 224 identifies the product shelf 410 or product stand 420 (collectively referred to as a display unit for simplicity) from the space in the line of sight direction in the world coordinate system, as explained in relation to Figure 3, and the layout information in the world coordinate system. Based on the space occupied by the display unit, the space included in both the line of sight and the space occupied by the display unit is identified in the world coordinate system ... the target area identification unit 224 can identify the space within the person's field of view as the gaze space in the world coordinate system, based on the person's position coordinates in the world coordinate system and the layout information of the three-dimensional object. If the field of view is not obstructed by other three-dimensional objects, the specified space can be designated as the gaze space” and in paragraph [0106], “the gaze identification unit 204 can identify the space in the direction of a person's gaze in the world coordinate system based on the position and field of view of the imaging device 100 that captured the frame image 1100a. Furthermore, the target area identification unit 224 can identify the gaze space in the world coordinate system ... The target area identification unit 224 can identify the area on the frame image 1100b in which the gaze space is captured as a target area, based on the position and field of view in the world coordinate system of the imaging device 100 that captured the frame image 1100b, and the gaze space in the world coordinate system. As a result, the target area identification unit 224 can identify the area on the frame image 1100b that corresponds to the space where an object such as a product that a person is actually looking at exists as the target area.”
Yonaha additionally discloses in paragraph [0056], “the gaze identification unit 204 identifies the angle of the person's gaze direction in three-dimensional space, for example, the angle of the person's gaze direction in three-dimensional space relative to the direction from the person's position in three-dimensional space to the position of the imaging device 100.”
The independent claims do not specifically define the term “characteristic information,” however the independent claims indicate that the characteristic information is related to an angle and a distance at which the target object can be gazed.
Yonaha discloses that the object must be within a field of view of the person, within the angle of the person’s gaze in three-dimensional space and is not obstructed. The field of view is a measurement of distance and combined with the angle comprises the information in which the target object can be gazed at. As the person moves closer or more distant from the object, the field of view and therefore the distance and angle will change and so will the distance and angle at which the target object can be gazed at.
Therefore, the combination of references discloses the limitation, “acquire characteristic information related to an angle and a distance at which the target object can be gazed at” as recited in the independent claims.
Claim Objections
Claims 1, 15 and 16 are objected to because of the following informalities: Claims 1, 15 and 16 recite the limitation, “determine whether or not the person has gazed at the target object” in lines 9, 8 and 9, respectively of the claims. This limitation appears to contain a typographical error and should recite, i.e., “determine whether or not a person has gazed at the target object.” Appropriate correction is required.
Claims 1, 15 and 16 are objected to because of the following informalities: Claims 1, 15 and 16 recite the limitation, “a joint point of a person” in lines 10, 9 and 10 respectively of the claims. This limitation appears to contain a typographical error and should recite, i.e., “a joint point of the person.” Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claim(s) 1-8, 12 and 14-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yonaha, Japanese Patent Publication JP 2010268158A, hereinafter, “Yonaha”, and further in view of Yumiya H, Deguchi D, Kawanishi Y, Murase H. End-to-End Gaze Grounding of a Person Pictured from Behind. In VISIGRAPP (5: VISAPP) 2023 Feb 19-21 (pp. 898-905), hereinafter, “Yumiya”.
As per claim 1, Yonaha discloses a processing apparatus comprising:
one or more memories storing instructions; and one or more processors executing the instructions (Yonaha, ¶0029, The storage medium 90 stores a program for the image processing device 110. The program stored in the storage medium 90 is provided to an electronic information processing device, such as a computer ... The CPU of the computer operates according to the content of the program and controls various parts of the computer) to:
(1) acquire a video of a place where a target object is located (Yonaha, ¶0019, The imaging device 100a images the store space 160. The imaging device 100a captures images of people 164a and 164b in the store space 160, along with product shelves 162a and 162b in the store space 160, and generates a video; Yonaha, ¶0027, the image processing device 110 highlights the area of the object of interest; Yonaha, ¶0036, the target area identification unit 224 may identify the line of sight area by extracting specific objects that exist … a specific object could be a specific product within a store);
(2) acquire characteristic information related to an angle and a distance at which the target object can be gazed at (Yonaha, ¶0067, the imaging range information storage unit 240 stores imaging range information that specifies the imaging range that each of the multiple imaging devices 100 will capture. As a result, the target area identification unit 224 can exclude areas of space in the direction of the person's line of sight from the line of sight area 330 where no objects exist, and can identify the image area in which the object seen by the person was captured; Yonaha, ¶0069, Based on the imaging range information described in relation to Figure 4, the target area identification unit 224 identifies the line-of-sight area 332, which is the image area in which the object seen by the person is captured, as the target area by excluding the area of space in which no object exists; Yonaha, ¶0070, the target area identification unit 224 identifies the product shelf 410 or product stand 420(collectively referred to as a display unit for simplicity) from the space in the line of sight direction in the world coordinate system, as explained in relation to Figure 3, and the layout information in the world coordinate system. Based on the space occupied by the display unit, the space included in both the line of sight and the space occupied by the display unit is identified in the world coordinate system ... the target area identification unit 224 can identify the space within the person's field of view as the gaze space in the world coordinate system, based on the person's position coordinates in the world coordinate system and the layout information of the three-dimensional object. If the field of view is not obstructed by other three-dimensional objects, the specified space can be designated as the gaze space; Yonaha, ¶0106, the gaze identification unit 204 can identify the space in the direction of a person's gaze in the world coordinate system based on the position and field of view of the imaging device 100 that captured the frame image 1100a. Furthermore, the target area identification unit 224 can identify the gaze space in the world coordinate system ... The target area identification unit 224 can identify the area on the frame image 1100b in which the gaze space is captured as a target area, based on the position and field of view in the world coordinate system of the imaging device 100 that captured the frame image 1100b, and the gaze space in the world coordinate system. As a result, the target area identification unit 224 can identify the area on the frame image 1100b that corresponds to the space where an object such as a product that a person is actually looking at exists as the target area; Yonaha, ¶0056, the gaze identification unit 204 identifies the angle of the person's gaze direction in three-dimensional space, for example, the angle of the person's gaze direction in three-dimensional space relative to the direction from the person's position in three-dimensional space to the position of the imaging device 100);
(3) set a gaze region, which is a region where the target object can be gazed at, based on the characteristic information that has been acquired (Yonaha, ¶0069, Based on the imaging range information described in relation to Figure 4, the target area identification unit 224 identifies the line-of-sight area 332, which is the image area in which the object seen by the person is captured, as the target area by excluding the area of space in which no object exists; Yonaha, ¶0070, the target area identification unit 224 identifies the product shelf 410 or product stand 420(collectively referred to as a display unit for simplicity) from the space in the line of sight direction in the world coordinate system, as explained in relation to Figure 3, and the layout information in the world coordinate system. Based on the space occupied by the display unit, the space included in both the line of sight and the space occupied by the display unit is identified in the world coordinate system ... the target area identification unit 224 can identify the space within the person's field of view as the gaze space in the world coordinate system, based on the person's position coordinates in the world coordinate system and the layout information of the three-dimensional object. If the field of view is not obstructed by other three-dimensional objects, the specified space can be designated as the gaze space); and
(4) determine whether or not the person has gazed at the target object (Yonaha, ¶0044, the gaze identification unit 204 may identify the gaze direction of the person indicated by the object identified by the feature area identification unit 202. Therefore, the image processing device 110 can reduce the amount of image data while maintaining the amount of image information of the product that the person is paying attention to and the image information of the person; Yonaha, ¶0045, The time-determining unit 226 determines the gaze time, which is the length of time the person was facing the direction of their gaze. For example, gaze time may be the length of time that a person continuously faced the direction of their gaze ... The duration for which the person's gaze was continuously directed in the above-mentioned direction does not mean the duration for which the person's gaze was perfectly aligned with the above-mentioned direction, but rather the duration for which the person's gaze was directed in a direction that was approximately aligned with the above-mentioned direction. In other words, the gaze duration may be the length of time during which the angle of change in the direction of gaze is smaller than a predetermined value; Yonaha, ¶0046, the image processing unit 230 may emphasize the gaze area more as the gaze duration increases).
Yonaha further discloses (Yonaha, ¶0052, if the eye object is located to the right among the head objects, the gaze identification unit 204 can identify the direction pointing to the right in the frame image 300 as the gaze direction. Furthermore, if the head object includes specific objects that can serve as landmarks within the head object, such as a mouth, nose, or ears, the gaze identification unit 204 can also determine the direction of the gaze based on the positional relationship between the specific object and the eye object; Yonaha, ¶0088, One example of an indicator used to calculate the degree of interest is the angle between the direction of gaze and the direction of the human body; Yonaha, ¶0090, the person in the human object 850b deliberately turns their head in a direction different from the direction their body is facing, looking in the direction where the product 860 is located. In this case, the person represented by the person object 850b is likely to be interested in the product 860. Thus, when customers find a product that interests them, they often turn their heads towards it before turning their bodies. The image processing unit 230 calculates a higher level of interest the larger the angle between the body direction and the line of sight direction; Yonaha, ¶0093, if a person changes their direction of movement towards a particular product while looking at it, it can be inferred that the person is showing a stronger interest in that product. In this case, as time passes, the magnitude of the angle between the direction of movement and the direction of line of sight decreases over time. Since the image processing unit 230 calculates the level of interest from the magnitude of the temporal change in the angle, it can appropriately calculate the level of interest), but does not explicitly disclose the following limitations as further recited however Yumiya discloses
(4) determine whether or not the person has gazed at the target object, based on information on a joint point of a person that is present in the gaze region that has been set (Yumiya, pages 898-899, 1 Introduction, In this study, we propose an end-to-end method to estimate the area of the object at which a person is gazing from their posture ... human posture is defined as a set of 3D locations of body joints calculated by 3D pose estimation. In the proposed method, a posture is embedded into a posture feature space. The feature space is trained by deep metric learning to differentiate poses ... a likelihood map associated with the scene is generated from the embedded posture feature. By referring to the object location in the scene, the method aggregates likelihoods within each target object region. Finally, the region with the highest likelihood is selected as the region of the object of the person’s gaze … The method generates a likelihood map from posture information and associates the map with regions of objects ... We provide a method to aggregate likelihood by referring to object locations; Yumiya, page 900, 3 Estimating Gaze Object Area from Behind, To associate the gaze area with an actual object in the real world, we propose an end-to-end method that generates a likelihood map of the gaze area for a given posture and aggregates likelihoods within each object region to obtain object-wise likelihoods … we consider estimating the gaze object area by focusing on differences in posture … The Likelihood Aggregation process calculates the object-wise gazed likelihood from the likelihood map in reference to object locations; Yumiya, page 901, Figure 2: The architecture of the proposed model; Yumiya, page 901, 3.1 Posture Embedding Encoder Module, The input of the encoder is 21 three-dimensional coordinates of human joints that is, it is a 63-dimensional vector ... In the scene, there are several objects that people might be expected to gaze at. We assign object IDs for each object; Yumiya, page 901-902, 3.2 Likelihood Map Generator Module).
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of Yumiya and Yonaha because they are in the same field of endeavor. One skilled in the art would have been motivated to include the joint locations of Yumiya in place of the landmark features and gaze /body angle of interest as taught by Yonaha as an alternative means to determine the gaze target of a person (Yumiya, pages 898-899, 1 Introduction).
As pe claim 2, Yonaha and Yumiya disclose the processing apparatus according to claim 1. Yumiya discloses wherein the one or more processors perform correction of matching the gaze region to a position where the target object is located in the video based on height information of the person (Yumiya, page 900, 3 Estimating Gaze Object Area from Behind, When we analyze the postures more deeply, as shown in Figs. 1 (b) and (c), we can note some differences between individuals. In the figure, a different person is looking at an object placed at the lower left; they are in different postures even though they are looking at the same area. To compensate for these differences, we introduce a deep metric learning technique into the Posture Embedding Encoder module; Yumiya, page 899, Figure 1(b), Lower left (Person A); Figure 1(c), Lower left (Person B) 2 people of different heights looking at same object on shelf). The motivation would be the same as above in claim 1.
As per claim 3, Yonaha and Yumiya disclose the processing apparatus according to claim 1, wherein the gaze region is a partial region in a three-dimensional space set with reference to a position of the target object and is determined by a distance from the target object to the person and an angle of the person with respect to the target object (Yonaha, ¶0069, Based on the imaging range information described in relation to Figure 4, the target area identification unit 224 identifies the line-of-sight area 332, which is the image area in which the object seen by the person is captured, as the target area by excluding the area of space in which no object exists in the direction of the person's line of sight from the line-of-sight area 330 described in relation to Figure 3. This allows for a more limited viewing range; Yonaha, ¶0070, the target area identification unit 224 identifies the product shelf 410 or product stand 420 (collectively referred to as a display unit for simplicity) from the space in the line of sight direction in the world coordinate system, as explained in relation to Figure 3, and the layout information in the world coordinate system. Based on the space occupied by the display unit, the space included in both the line of sight and the space occupied by the display unit is identified in the world coordinate system ... the target area identification unit 224 can identify the space within the person's field of view as the gaze space in the world coordinate system, based on the person's position coordinates in the world coordinate system and the layout information of the three-dimensional object. If the field of view is not obstructed by other three-dimensional objects, the specified space can be designated as the gaze space).
As per claim 4, Yonaha and Yumiya disclose the processing apparatus according to claim 1, wherein the one or more processors acquire location information related to a location of the target object (Yumiya, page 901, Figure 2, Object Region; Yumiya, page 904, Figure 6: Result of a likelihood map estimated for person A looking at upper left; Yumiya, page 904, Figure 7, Result of a likelihood map estimated for person A looking at lower left; Yumiya, page 899, 1 Introduction, posture differences depending on the gaze target ... a likelihood map associated with the scene is generated from the embedded posture feature. By referring to the object location in the scene, the method aggregates likelihoods within each target object region. Finally, the region with the highest likelihood is selected as the region of the object of the person’s gaze; Yumiya, page 900, 3 Estimating Gaze Object Area from Behind, To associate the gaze area with an actual object in the real world, we propose an end-to-end method that generates a likelihood map of the gaze area for a given posture and aggregates likelihoods within each object region to obtain object-wise likelihoods; Yumiya, page 903, 4.2 Experimental Settings, First, the proposed encoder module is applied to posture (3D coordinates of human joints) to obtain posture features in the feature space. Then, deconvolutional neural network is applied to the embedded features to reconstruct a likelihood map corresponding to the subject’s gaze area. Finally, object-wise likelihoods are aggregated for every object region referring to object locations; Yonaha, ¶0065, Figure 4 shows a two-dimensional example of the layout information stored in the imaging range information storage unit 240. The imaging range information storage unit 240 stores, as layout information of the store space 160, the range in the store interior plane 400 where the product shelves 410, product stands 420, and cash register 430 are located, as shown in this figure. Furthermore, the layout information may include height information for three-dimensional objects such as product shelves 410, product displays 420, and cash registers 430. Thus, the imaging range information storage unit 240 may store layout information that shows the three-dimensional layout of the store space 160; Yonaha, ¶0066, the imaging range information storage unit 240 stores information on the imaging direction and field of view captured by the imaging device 100, as well as position information indicating the position of the imaging device 100. Examples of location information include three-dimensional coordinate values within the store space160. Examples of imaging direction and field of view information include three-dimensional vector information in the store space 160. The imaging direction, field of view, position information, and layout information of the imaging device100 may be defined in a world coordinate system that represents the entire space of the store space160. The imaging range information storage unit 240 stores imaging direction and field of view information, position information, and layout information, which allow the imaging range to be determined within the store space 160).
As per claim 5, Yonaha and Yumiya disclose the processing apparatus according to claim 4. Yumiya discloses wherein the one or more processors correct the gaze region based on the location information (Yumiya, page 904, Figure 6: Result of a likelihood map estimated for person A looking at upper left; Yumiya, page 904, Figure 7, Result of a likelihood map estimated for person A looking at lower left; Yumiya, page 899, 1 Introduction, posture differences depending on the gaze target ... a likelihood map associated with the scene is generated from the embedded posture feature. By referring to the object location in the scene, the method aggregates likelihoods within each target object region. Finally, the region with the highest likelihood is selected as the region of the object of the person’s gaze; Yumiya, page 900, 3 Estimating Gaze Object Area from Behind, To associate the gaze area with an actual object in the real world, we propose an end-to-end method that generates a likelihood map of the gaze area for a given posture and aggregates likelihoods within each object region to obtain object-wise likelihoods … When we analyze the postures more deeply, as shown in Figs. 1 (b) and (c), we can note some differences between individuals. In the figure, a different person is looking at an object placed at the lower left; they are in different postures even though they are looking at the same area. To compensate for these differences, we introduce a deep metric learning technique into the Posture Embedding Encoder module). The motivation would be the same as above in claim 1.
As per claim 6, Yonaha and Yumiya disclose the processing apparatus according to claim 4, wherein the location information is information including at least one of information on a position where the target object is located, an illumination condition, and a position of a shielding object around the target object (Yumiya, page 901, Figure 2, Object Region; Yonaha, ¶0065, Figure 4 shows a two-dimensional example of the layout information stored in the imaging range information storage unit 240. The imaging range information storage unit 240 stores, as layout information of the store space 160, the range in the store interior plane 400 where the product shelves 410, product stands 420, and cash register 430 are located, as shown in this figure. Furthermore, the layout information may include height information for three-dimensional objects such as product shelves 410, product displays 420, and cash registers 430. Thus, the imaging range information storage unit 240 may store layout information that shows the three-dimensional layout of the store space 160).
As per claim 7, Yonaha and Yumiya discloses the processing apparatus according to claim 6, wherein the gaze region is corrected so as not to overlap with the position of the shielding object based on the position of the shielding object (Yonaha, ¶0070, the target area identification unit 224 identifies the product shelf 410 or product stand 420 (collectively referred to as a display unit for simplicity) from the space in the line of sight direction in the world coordinate system, as explained in relation to Figure 3, and the layout information in the world coordinate system. Based on the space occupied by the display unit, the space included in both the line of sight and the space occupied by the display unit is identified in the world coordinate system ... due to other three-dimensional objects obstructing a person's field of vision, only a portion of a specific space may be visible to the person. In this case, the target area identification unit 224 can identify the space within the person's field of view as the gaze space in the world coordinate system, based on the person's position coordinates in the world coordinate system and the layout information of the three-dimensional object).
As per claim 8, Yonaha and Yumiya disclose the processing apparatus according to claim 1, wherein the one or more processors acquire the video in a plurality of frames, detect a region of a person in each of the plurality of frames, estimate a posture from the detected region of the person, and estimate information on a joint point of the person based on the posture (Yonaha, ¶0047, the gaze identification unit 204 identifies the gaze direction of a person captured in the video, and the gaze region analysis unit 220 identifies the gaze region in the video based on the gaze direction; Yonaha, ¶0048, The frame image 300 includes a person object 350 and a product stand object 360; Yonaha, ¶0049, The feature region identification unit 202 extracts the region in which a person is captured from the image content of the frame image 300, thereby extracting the person object 350; Yonaha, ¶0050, The feature region identification unit 202 identifies the head region 310, which includes the object of the person's head, as a feature region. Furthermore, the feature region identification unit 202 identifies the body region 320, which includes the object of the person's body, as a feature region; Yonaha, ¶0051, The gaze identification unit 204 identifies the gaze direction 380 of the person in the person object 350 from the area of the head object in the person object 350. For example, the gaze identification unit 204 may determine the gaze direction based on the position of the eye object within the head object, the positional relationship between the eye object and other objects, etc.; Yonaha, ¶0052, For example, if the eye object is located to the right among the head objects, the gaze identification unit204 can identify the direction pointing to the right in the frame image 300 as the gaze direction. Furthermore, if the head object includes specific objects that can serve as landmarks within the head object, such as a mouth, nose, or ears, the gaze identification unit 204 can also determine the direction of the gaze based on the positional relationship between the specific object and the eye object).
As per claim 12, Yonaha and Yumiya disclose the processing apparatus according to claim 1, wherein, in a case in which the one or more processors determine that the person within the gaze region has gazed at the target object, the one or more processors determine whether or not the person is interested in the target object, based on information on a time during which the person is gazing at the target object (Yonaha, ¶0026, The image processing device 110 then identifies the object of interest, which is the image of the object that the person 164 is focusing on, based on the direction of the person 164's gaze within the image area of the frame image. The image processing device 110 performs processing to highlight the region of the identified object of interest; Yonaha, ¶0044, the gaze identification unit 204 may identify the gaze direction of the person indicated by the object identified by the feature area identification unit 202; Yonaha, ¶0045, The time-determining unit 226 determines the gaze time, which is the length of time the person was facing the direction of their gaze. For example, gaze time may be the length of time that a person continuously faced the direction of their gaze ... The duration for which the person's gaze was continuously directed in the above-mentioned direction does not mean the duration for which the person's gaze was perfectly aligned with the above-mentioned direction, but rather the duration for which the person's gaze was directed in a direction that was approximately aligned with the above-mentioned direction. In other words, the gaze duration may be the length of time during which the angle of change in the direction of gaze is smaller than a predetermined value; Yonaha, ¶0046, the image processing unit 230 may emphasize the gaze area more as the gaze duration increases).
As per claim 14, Yonaha and Yumiya disclose the processing apparatus according to claim 1, wherein the characteristic information is information including any one of a size of a character on the target object, a contrast difference between a background and a character, a font used, and a shape of the target object (Yonaha, ¶0043, The feature region identification unit 202 identifies a feature region, which is a characteristic region in the image ... the feature region identification unit 202 may identify a region containing an object that conforms to predetermined conditions as a feature region; Yonaha, ¶0044, the gaze identification unit 204 may identify the gaze direction of the person indicated by the object identified by the feature area identification unit 202. Therefore, the image processing device 110 can reduce the amount of image data while maintaining the amount of image information of the product that the person is paying attention to and the image information of the person [indicative of a contrast difference between the product and the background / rest of image frame]).
As per claim 15, Yonaha discloses a control method of a processing apparatus, the method comprising:
acquiring a video of a place where a target object is located (Yonaha, ¶0019, The imaging device 100a images the store space 160. The imaging device 100a captures images of people 164a and 164b in the store space 160, along with product shelves 162a and 162b in the store space 160, and generates a video; Yonaha, ¶0027, the image processing device 110 highlights the area of the object of interest; Yonaha, ¶0036, the target area identification unit 224 may identify the line of sight area by extracting specific objects that exist … a specific object could be a specific product within a store);
acquiring characteristic information related to an angle and a distance at which the target object can be gazed at (Yonaha, ¶0067, the imaging range information storage unit 240 stores imaging range information that specifies the imaging range that each of the multiple imaging devices 100 will capture. As a result, the target area identification unit 224 can exclude areas of space in the direction of the person's line of sight from the line of sight area 330 where no objects exist, and can identify the image area in which the object seen by the person was captured; Yonaha, ¶0069, Based on the imaging range information described in relation to Figure 4, the target area identification unit 224 identifies the line-of-sight area 332, which is the image area in which the object seen by the person is captured, as the target area by excluding the area of space in which no object exists; Yonaha, ¶0070, the target area identification unit 224 identifies the product shelf 410 or product stand 420(collectively referred to as a display unit for simplicity) from the space in the line of sight direction in the world coordinate system, as explained in relation to Figure 3, and the layout information in the world coordinate system. Based on the space occupied by the display unit, the space included in both the line of sight and the space occupied by the display unit is identified in the world coordinate system ... the target area identification unit 224 can identify the space within the person's field of view as the gaze space in the world coordinate system, based on the person's position coordinates in the world coordinate system and the layout information of the three-dimensional object. If the field of view is not obstructed by other three-dimensional objects, the specified space can be designated as the gaze space; Yonaha, ¶0106, the gaze identification unit 204 can identify the space in the direction of a person's gaze in the world coordinate system based on the position and field of view of the imaging device 100 that captured the frame image 1100a. Furthermore, the target area identification unit 224 can identify the gaze space in the world coordinate system ... The target area identification unit 224 can identify the area on the frame image 1100b in which the gaze space is captured as a target area, based on the position and field of view in the world coordinate system of the imaging device 100 that captured the frame image 1100b, and the gaze space in the world coordinate system. As a result, the target area identification unit 224 can identify the area on the frame image 1100b that corresponds to the space where an object such as a product that a person is actually looking at exists as the target area; Yonaha, ¶0056, the gaze identification unit 204 identifies the angle of the person's gaze direction in three-dimensional space, for example, the angle of the person's gaze direction in three-dimensional space relative to the direction from the person's position in three-dimensional space to the position of the imaging device 100);
setting a gaze region, which is a region where the target object can be gazed at, based on the characteristic information that has been acquired (Yonaha, ¶0069, Based on the imaging range information described in relation to Figure 4, the target area identification unit 224 identifies the line-of-sight area 332, which is the image area in which the object seen by the person is captured, as the target area by excluding the area of space in which no object exists; Yonaha, ¶0070, the target area identification unit 224 identifies the product shelf 410 or product stand 420(collectively referred to as a display unit for simplicity) from the space in the line of sight direction in the world coordinate system, as explained in relation to Figure 3, and the layout information in the world coordinate system. Based on the space occupied by the display unit, the space included in both the line of sight and the space occupied by the display unit is identified in the world coordinate system ... the target area identification unit 224 can identify the space within the person's field of view as the gaze space in the world coordinate system, based on the person's position coordinates in the world coordinate system and the layout information of the three-dimensional object. If the field of view is not obstructed by other three-dimensional objects, the specified space can be designated as the gaze space); and
determining whether or not the person has gazed at the target object (Yonaha, ¶0044, the gaze identification unit 204 may identify the gaze direction of the person indicated by the object identified by the feature area identification unit 202. Therefore, the image processing device 110 can reduce the amount of image data while maintaining the amount of image information of the product that the person is paying attention to and the image information of the person; Yonaha, ¶0045, The time-determining unit 226 determines the gaze time, which is the length of time the person was facing the direction of their gaze. For example, gaze time may be the length of time that a person continuously faced the direction of their gaze ... The duration for which the person's gaze was continuously directed in the above-mentioned direction does not mean the duration for which the person's gaze was perfectly aligned with the above-mentioned direction, but rather the duration for which the person's gaze was directed in a direction that was approximately aligned with the above-mentioned direction. In other words, the gaze duration may be the length of time during which the angle of change in the direction of gaze is smaller than a predetermined value; Yonaha, ¶0046, the image processing unit 230 may emphasize the gaze area more as the gaze duration increases).
Yonaha further discloses (Yonaha, ¶0052, if the eye object is located to the right among the head objects, the gaze identification unit 204 can identify the direction pointing to the right in the frame image 300 as the gaze direction. Furthermore, if the head object includes specific objects that can serve as landmarks within the head object, such as a mouth, nose, or ears, the gaze identification unit 204 can also determine the direction of the gaze based on the positional relationship between the specific object and the eye object; Yonaha, ¶0088, One example of an indicator used to calculate the degree of interest is the angle between the direction of gaze and the direction of the human body; Yonaha, ¶0090, the person in the human object 850b deliberately turns their head in a direction different from the direction their body is facing, looking in the direction where the product 860 is located. In this case, the person represented by the person object 850b is likely to be interested in the product 860. Thus, when customers find a product that interests them, they often turn their heads towards it before turning their bodies. The image processing unit 230 calculates a higher level of interest the larger the angle between the body direction and the line of sight direction; Yonaha, ¶0093, if a person changes their direction of movement towards a particular product while looking at it, it can be inferred that the person is showing a stronger interest in that product. In this case, as time passes, the magnitude of the angle between the direction of movement and the direction of line of sight decreases over time. Since the image processing unit 230 calculates the level of interest from the magnitude of the temporal change in the angle, it can appropriately calculate the level of interest), but does not explicitly disclose the following limitations as further recited however Yumiya discloses
determining whether or not the person has gazed at the target object, based on information on a joint point of a person who is present in the gaze region that has been set (Yumiya, pages 898-899, 1 Introduction, In this study, we propose an end-to-end method to estimate the area of the object at which a person is gazing from their posture ... human posture is defined as a set of 3D locations of body joints calculated by 3D pose estimation. In the proposed method, a posture is embedded into a posture feature space. The feature space is trained by deep metric learning to differentiate poses ... a likelihood map associated with the scene is generated from the embedded posture feature. By referring to the object location in the scene, the method aggregates likelihoods within each target object region. Finally, the region with the highest likelihood is selected as the region of the object of the person’s gaze … The method generates a likelihood map from posture information and associates the map with regions of objects ... We provide a method to aggregate likelihood by referring to object locations; Yumiya, page 900, 3 Estimating Gaze Object Area from Behind, To associate the gaze area with an actual object in the real world, we propose an end-to-end method that generates a likelihood map of the gaze area for a given posture and aggregates likelihoods within each object region to obtain object-wise likelihoods … we consider estimating the gaze object area by focusing on differences in posture … The Likelihood Aggregation process calculates the object-wise gazed likelihood from the likelihood map in reference to object locations; Yumiya, page 901, Figure 2: The architecture of the proposed model; Yumiya, page 901, 3.1 Posture Embedding Encoder Module, The input of the encoder is 21 three-dimensional coordinates of human joints that is, it is a 63-dimensional vector ... In the scene, there are several objects that people might be expected to gaze at. We assign object IDs for each object; Yumiya, page 901-902, 3.2 Likelihood Map Generator Module).
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of Yumiya and Yonaha because they are in the same field of endeavor. One skilled in the art would have been motivated to include the joint locations of Yumiya in place of the landmark features and gaze /body angle of interest as taught by Yonaha as an alternative means to determine the gaze target of a person (Yumiya, pages 898-899, 1 Introduction).
As per claim 16, Yonaha discloses a non-transitory computer-readable storage medium configured to store a computer program comprising instructions (Yonaha, ¶0029, The storage medium 90 stores a program for the image processing device 110. The program stored in the storage medium 90 is provided to an electronic information processing device, such as a computer ... The CPU of the computer operates according to the content of the program and controls various parts of the computer) for executing processes comprising:
acquiring a video of a place where a target object is located (Yonaha, ¶0019, The imaging device 100a images the store space 160. The imaging device 100a captures images of people 164a and 164b in the store space 160, along with product shelves 162a and 162b in the store space 160, and generates a video; Yonaha, ¶0027, the image processing device 110 highlights the area of the object of interest; Yonaha, ¶0036, the target area identification unit 224 may identify the line of sight area by extracting specific objects that exist … a specific object could be a specific product within a store);
acquiring characteristic information related to an angle and a distance at which the target object can be gazed at (Yonaha, ¶0067, the imaging range information storage unit 240 stores imaging range information that specifies the imaging range that each of the multiple imaging devices 100 will capture. As a result, the target area identification unit 224 can exclude areas of space in the direction of the person's line of sight from the line of sight area 330 where no objects exist, and can identify the image area in which the object seen by the person was captured; Yonaha, ¶0069, Based on the imaging range information described in relation to Figure 4, the target area identification unit 224 identifies the line-of-sight area 332, which is the image area in which the object seen by the person is captured, as the target area by excluding the area of space in which no object exists; Yonaha, ¶0070, the target area identification unit 224 identifies the product shelf 410 or product stand 420(collectively referred to as a display unit for simplicity) from the space in the line of sight direction in the world coordinate system, as explained in relation to Figure 3, and the layout information in the world coordinate system. Based on the space occupied by the display unit, the space included in both the line of sight and the space occupied by the display unit is identified in the world coordinate system ... the target area identification unit 224 can identify the space within the person's field of view as the gaze space in the world coordinate system, based on the person's position coordinates in the world coordinate system and the layout information of the three-dimensional object. If the field of view is not obstructed by other three-dimensional objects, the specified space can be designated as the gaze space; Yonaha, ¶0106, the gaze identification unit 204 can identify the space in the direction of a person's gaze in the world coordinate system based on the position and field of view of the imaging device 100 that captured the frame image 1100a. Furthermore, the target area identification unit 224 can identify the gaze space in the world coordinate system ... The target area identification unit 224 can identify the area on the frame image 1100b in which the gaze space is captured as a target area, based on the position and field of view in the world coordinate system of the imaging device 100 that captured the frame image 1100b, and the gaze space in the world coordinate system. As a result, the target area identification unit 224 can identify the area on the frame image 1100b that corresponds to the space where an object such as a product that a person is actually looking at exists as the target area; Yonaha, ¶0056, the gaze identification unit 204 identifies the angle of the person's gaze direction in three-dimensional space, for example, the angle of the person's gaze direction in three-dimensional space relative to the direction from the person's position in three-dimensional space to the position of the imaging device 100.);
setting a gaze region, which is a region where the target object can be gazed at, based on the characteristic information that has been acquired (Yonaha, ¶0069, Based on the imaging range information described in relation to Figure 4, the target area identification unit 224 identifies the line-of-sight area 332, which is the image area in which the object seen by the person is captured, as the target area by excluding the area of space in which no object exists; Yonaha, ¶0070, the target area identification unit 224 identifies the product shelf 410 or product stand 420(collectively referred to as a display unit for simplicity) from the space in the line of sight direction in the world coordinate system, as explained in relation to Figure 3, and the layout information in the world coordinate system. Based on the space occupied by the display unit, the space included in both the line of sight and the space occupied by the display unit is identified in the world coordinate system ... the target area identification unit 224 can identify the space within the person's field of view as the gaze space in the world coordinate system, based on the person's position coordinates in the world coordinate system and the layout information of the three-dimensional object. If the field of view is not obstructed by other three-dimensional objects, the specified space can be designated as the gaze space); and
determining whether or not the person has gazed at the target object (Yonaha, ¶0044, the gaze identification unit 204 may identify the gaze direction of the person indicated by the object identified by the feature area identification unit 202. Therefore, the image processing device 110 can reduce the amount of image data while maintaining the amount of image information of the product that the person is paying attention to and the image information of the person; Yonaha, ¶0045, The time-determining unit 226 determines the gaze time, which is the length of time the person was facing the direction of their gaze. For example, gaze time may be the length of time that a person continuously faced the direction of their gaze ... The duration for which the person's gaze was continuously directed in the above-mentioned direction does not mean the duration for which the person's gaze was perfectly aligned with the above-mentioned direction, but rather the duration for which the person's gaze was directed in a direction that was approximately aligned with the above-mentioned direction. In other words, the gaze duration may be the length of time during which the angle of change in the direction of gaze is smaller than a predetermined value; Yonaha, ¶0046, the image processing unit 230 may emphasize the gaze area more as the gaze duration increases).
Yonaha further discloses (Yonaha, ¶0052, if the eye object is located to the right among the head objects, the gaze identification unit 204 can identify the direction pointing to the right in the frame image 300 as the gaze direction. Furthermore, if the head object includes specific objects that can serve as landmarks within the head object, such as a mouth, nose, or ears, the gaze identification unit 204 can also determine the direction of the gaze based on the positional relationship between the specific object and the eye object; Yonaha, ¶0088, One example of an indicator used to calculate the degree of interest is the angle between the direction of gaze and the direction of the human body; Yonaha, ¶0090, the person in the human object 850b deliberately turns their head in a direction different from the direction their body is facing, looking in the direction where the product 860 is located. In this case, the person represented by the person object 850b is likely to be interested in the product 860. Thus, when customers find a product that interests them, they often turn their heads towards it before turning their bodies. The image processing unit 230 calculates a higher level of interest the larger the angle between the body direction and the line of sight direction; Yonaha, ¶0093, if a person changes their direction of movement towards a particular product while looking at it, it can be inferred that the person is showing a stronger interest in that product. In this case, as time passes, the magnitude of the angle between the direction of movement and the direction of line of sight decreases over time. Since the image processing unit 230 calculates the level of interest from the magnitude of the temporal change in the angle, it can appropriately calculate the level of interest), but does not explicitly disclose the following limitations as further recited however Yumiya discloses
determining whether or not the person has gazed at the target object, based on information on a joint point of a person who is present in the gaze region that has been set (Yumiya, pages 898-899, 1 Introduction, In this study, we propose an end-to-end method to estimate the area of the object at which a person is gazing from their posture ... human posture is defined as a set of 3D locations of body joints calculated by 3D pose estimation. In the proposed method, a posture is embedded into a posture feature space. The feature space is trained by deep metric learning to differentiate poses ... a likelihood map associated with the scene is generated from the embedded posture feature. By referring to the object location in the scene, the method aggregates likelihoods within each target object region. Finally, the region with the highest likelihood is selected as the region of the object of the person’s gaze … The method generates a likelihood map from posture information and associates the map with regions of objects ... We provide a method to aggregate likelihood by referring to object locations; Yumiya, page 900, 3 Estimating Gaze Object Area from Behind, To associate the gaze area with an actual object in the real world, we propose an end-to-end method that generates a likelihood map of the gaze area for a given posture and aggregates likelihoods within each object region to obtain object-wise likelihoods … we consider estimating the gaze object area by focusing on differences in posture … The Likelihood Aggregation process calculates the object-wise gazed likelihood from the likelihood map in reference to object locations; Yumiya, page 901, Figure 2: The architecture of the proposed model; Yumiya, page 901, 3.1 Posture Embedding Encoder Module, The input of the encoder is 21 three-dimensional coordinates of human joints that is, it is a 63-dimensional vector ... In the scene, there are several objects that people might be expected to gaze at. We assign object IDs for each object; Yumiya, page 901-902, 3.2 Likelihood Map Generator Module).
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of Yumiya and Yonaha because they are in the same field of endeavor. One skilled in the art would have been motivated to include the joint locations of Yumiya in place of the landmark features and gaze /body angle of interest as taught by Yonaha as an alternative means to determine the gaze target of a person (Yumiya, pages 898-899, 1 Introduction).
Claim(s) 9-11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yonaha, Japanese Patent Publication JP 2010268158A, hereinafter, “Yonaha”, in view of Yumiya H, Deguchi D, Kawanishi Y, Murase H. End-to-End Gaze Grounding of a Person Pictured from Behind. In VISIGRAPP (5: VISAPP) 2023 Feb 19-21 (pp. 898-905), hereinafter, “Yumiya” as applied to claim 1 above, and further in view of Moon et al., U.S. Patent No. 7,742,623, hereinafter, “Moon”.
As per claim 9, Yonaha and Yumiya disclose the processing apparatus according to claim 1, wherein information on the joint point includes coordinate information on each joint of the person (Yumiya, page 901, Figure 2: The architecture of the proposed model; Yumiya, page 901, 3.1 Posture Embedding Encoder Module, The input of the encoder is 21 three-dimensional coordinates of human joints that is, it is a 63-dimensional vector).
Yonaha and Yumiya further discloses (Yonaha, ¶0052, if the eye object is located to the right among the head objects, the gaze identification unit204 can identify the direction pointing to the right in the frame image 300 as the gaze direction. Furthermore, if the head object includes specific objects that can serve as landmarks within the head object, such as a mouth, nose, or ears, the gaze identification unit 204 can also determine the direction of the gaze based on the positional relationship between the specific object and the eye object … even if the eye object is not included, the gaze identification unit 204 may identify the direction of the face as the gaze direction if it can determine the orientation of the face based on the positional relationship of the identified object) but do not explicitly disclose the following limitations as further recited however Moon discloses
wherein the information on the joint point includes coordinate information of a right eye, a left eye, a right ear, a left ear, and a nose of the person, (Moon, Figure 8; Moon, column 10, lines 15-26, FIG. 12 shows a facial feature localization training scheme 840 in an exemplary embodiment of the present invention. This exemplary training scheme aims to estimate the x (horizontal) shift, y (vertical) shift, the scale, and the orientation of the right eye within the standard facial feature window 406. Each eye image 421 is generated by cropping the standard facial feature window 406 of the right eye from the localized facial image 384. The facial landmark points of the face are assumed to be known, and the coordinates of the landmark points 657, after going through the localization based on the two-dimensional facial pose estimation 380 step, are available).
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of Moon with Yonaha and Yumiya because they are in the same field of endeavor. One skilled in the art would have been motivated to include the facial feature joint coordinates as taught by Moon in the system of Yonaha and Yumiya in order to provide a means to determine the gaze target of a person when facial features are available in addition to pose and posture (Moon, column 10, lines 15-26; Yonaha, ¶0052).
As per claim 10, Yonaha, Yumiya and Moon disclose the processing apparatus according to claim 9. Moon discloses wherein the one or more processors determine whether or not the person has gazed at the target object by performing (1) a first determination in which whether or not a center position is within the gaze region is determined, the center position being an average coordinate of coordinates of a right eye, a left eye, a right ear, a left ear, and a nose of the person (Moon, column 11, lines 25-42, FIG. 16 shows a function of the three-dimensional facial pose-dependent gaze direction estimation 955 step. The eye gaze 910 estimated from the eye gaze estimation 960 step is manifested by the movement of the iris, and is independent of the three-dimensional facial pose. Therefore, the two instances of the same eye gaze can actually point to a different gaze target 925 depending on the three-dimensional facial pose 391. In the figure, the top face shows the frontal pose, and the corresponding eye gaze reveals that the person is looking to the right. Therefore, the eye gaze 910 is the only element to estimate the gaze direction 901. In the bottom face, the face is pointing to the right. The eye gaze 910 of the person appears to be very similar to the first face, but needs to be interpreted differently from the case with the top face; the three-dimensional facial pose 391 should be additively incorporated to the final estimate of gaze direction; Moon, column 11, line 55 - column 12, line 13, FIG. 18 shows an exemplary embodiment of the three-dimensional facial pose-dependent gaze direction estimation 955 step ... a three-dimensional facial pose-dependent learning machine 958 that is trained for a particular three-dimensional facial pose 391. Once the eye gaze 910 and three-dimensional facial pose 391 are estimated from face view 342 ... the output gaze direction estimates are weighted to produce the gaze direction estimate; the weight 816 is proportional to a measure of similarity), and
a (2) second determination in which whether or not the person faces the target object is determined based on a positional relation between a right eye, a left eye, a right ear, and a left ear of the person (Moon, column 12, lines 14-37, FIG. 19 shows the person position estimation 725 and gaze target estimation 970 steps in an exemplary embodiment of the present invention. The person position estimation 725 step provides the world coordinate of the person ... The view-based person blob model 732 consists of an approximate shape of a body outline and the position of the head in each floor position. Once the body blob 730 is estimated, the head position 735 is located based on the model. The gaze direction estimation 950 step provides the gaze direction estimate 952, which is the orientation of gaze direction relative to the face-view camera 110. The visual target 920 depends on the face-view camera 110 orientation and the position of the person as shown in the figure; first the gaze direction 901 is interpreted for each face-view camera 110 using the gaze to target grid mapping 974 to estimate the gaze line 904. The coordinate of the gaze target is estimated by finding the intersection of the gaze line 904 with the visual target plane 921. The gaze line 904 is a line originating from the person's position having the same orientation as the estimated gaze direction). The motivation would be the same as above in claim 9.
As per claim 11, Yonaha, Yumiya and Moon disclose the processing apparatus according to claim 10. Moon discloses wherein, in a case in which a plurality of the target objects and the gaze regions is present and the center position of the person is present within a region where one gaze region and another gaze region partially overlap, the one or more processors determine which target object the person has gazed at, based on center position information of each of the gaze regions (Moon, column 12, lines 14-37, FIG. 19 shows the person position estimation 725 and gaze target estimation 970 steps in an exemplary embodiment of the present invention. The person position estimation 725 step provides the world coordinate of the person ... The view-based person blob model 732 consists of an approximate shape of a body outline and the position of the head in each floor position. Once the body blob 730 is estimated, the head position 735 is located based on the model. The gaze direction estimation 950 step provides the gaze direction estimate 952, which is the orientation of gaze direction relative to the face-view camera 110. The visual target 920 depends on the face-view camera 110 orientation and the position of the person as shown in the figure; first the gaze direction 901 is interpreted for each face-view camera 110 using the gaze to target grid mapping 974 to estimate the gaze line 904. The coordinate of the gaze target is estimated by finding the intersection of the gaze line 904 with the visual target plane 921. The gaze line 904 is a line originating from the person's position having the same orientation as the estimated gaze direction; Moon, column 12, lines 38-60, FIG. 20 shows the view selection 976 scheme in an exemplary embodiment of the present invention. When the system employs multiple face-view cameras, one person's face can appear to more than one face view 342. Because the accuracy of the gaze target estimation 970 greatly depends on the accuracy of the eye gaze estimation 960, it is crucial to choose the view that provides a better view of the face--more specifically, the eyes. The step chooses the best view of the face based on the person position 724 and the three-dimensional facial pose 391 of the person, because both the distance between the face and the camera and the three-dimensional facial pose relative to the camera affect the view. In the figure, the face of person 1 701 may appear to both the face view 1 343 and face view 2 344. The face view 1 343 is the selected view 977 for the person 1 701 based on the close distance to the face-view camera 1 111. The face of person 2 702 can appear to both the face view 2 344 and the face view 3 345. The person is slightly closer to the face-view camera 3 113, but is facing the face-view camera 2 112. Therefore, the face view 2 344 is the selected view 977). The motivation would be the same as above in claim 9.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yonaha, Japanese Patent Publication JP 2010268158A, hereinafter, “Yonaha”, in view of Yumiya H, Deguchi D, Kawanishi Y, Murase H. End-to-End Gaze Grounding of a Person Pictured from Behind. In VISIGRAPP (5: VISAPP) 2023 Feb 19-21 (pp. 898-905) , hereinafter, “Yumiya” as applied to claim 12 above, and further in view of Mihara et al., U.S. Publication No. 2017/0011260, hereinafter, “Mihara”.
As per claim 13, Yonaha and Yumiya disclose the processing apparatus according to claim 12, but do not explicitly disclose the following limitations as further recited however Mihara discloses wherein the one or more processors count the number of persons who are determined to be interested in the target object and display the number of counted persons on a display device (Mihara, ¶0036, FIG. 1 is a diagram illustrating the structure of a system according to an embodiment. As illustrated in FIG. 1, the system includes a camera 50 and an interest degree determination device 100. In addition, it is assumed that a product α, a product β, a product γ, and a product δ are displayed on a shelf 5 and the camera 50 is arranged such that a person who selects a product from the shelf 5 is included in an imaging range. Products other than the products α to δ may be displayed on the shelf 5; Mihara, ¶0045, FIG. 3 is a diagram illustrating an example of the data structure of the image database. As illustrated in FIG. 3, in the image database 141, a frame number, time, the number of persons detected, a person ID, a person region, the coordinates of the center of gravity, a frame, a plural flag, a singular flag, and a success flag; Mihara, ¶0049, The success flag is a flag indicating whether the detection of the gaze position of the person has succeeded; Mihara, ¶0050, The gaze target position database 142 stores information in which an image region of a virtual space is associated with a product included in the image region. FIG. 4 is a diagram illustrating an example of the data structure of the gaze target position database. As illustrated in FIG. 4, the gaze target position database 142 associates the image region of the virtual space with a product; Mihara, ¶0054, The person moving state check database 143a stores information about whether a person has approached a product. FIG. 6 is a diagram illustrating an example of the data structure of the person moving state check database. As illustrated in FIG. 6, the person moving state check database 143a has a table for each person ID. Here, for example, a table 20a for a person ID “1aaa” and a table 2Db for a person ID “1bbb” are illustrated; Mihara, ¶0055, In the tables 20a and 20b, time, movement, and a gaze target are associated with one another. The movement is information indicating whether a person corresponding to a person ID has approached a product which is a gaze target. When the person has approached the gaze target, the movement is “yes”. When the person has not approached the gaze target, the movement is “no”. The gaze target indicates a product at which the person corresponding to the person ID gazes. For example, a record in the first row of the table 20a indicates that a person corresponding to the person ID “1aaa” gazes at the product α; Mihara, ¶0068, FIG. 13 is a diagram illustrating an example of the data structure of the determination result database. As illustrated in FIG. 13, in the determination result database 144, time, the number of persons, a person ID, a gaze target product, and a spontaneity flag are associated with one another. The number of persons indicates the number of persons who have been subjected to a spontaneity determination process. The gaze target product indicates a product at which the person gazes. The spontaneity flag indicates whether the person spontaneously gazes at a product).
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of Mihara with Yonaha and Yumiya because they are in the same field of endeavor. One skilled in the art would have been motivated to include the count of the number of interested persons in a specific target object or product as taught by Mihara in the system of Yonaha and Yumiya in order to provide a means to determine the level of interest in a target object or product for marketing purposes (Mihara, ¶0003).
Claim(s) 17 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yonaha, Japanese Patent Publication JP 2010268158A, hereinafter, “Yonaha”, in view of Yumiya H, Deguchi D, Kawanishi Y, Murase H. End-to-End Gaze Grounding of a Person Pictured from Behind. In VISIGRAPP (5: VISAPP) 2023 Feb 19-21 (pp. 898-905), hereinafter, “Yumiya” as applied to claim 1 above, and further in view of Karuvath et al., U.S. Publication No. 2017/0308919, hereinafter, “Karuvath”.
As per claim 17, Yonaha and Yumiya disclose the processing apparatus according to Claim 1, but do not explicitly disclose the following limitations as further recited however Karuvath discloses wherein the characteristic information includes a size of a character on the target object (Karuvath, ¶0027, retrieve recommended items based on an identification of the person. The recommended items are displayed at a display located along the footpath of the target person ... this item could be information about a product (e.g., an advertisement or other information that informs the person about the product); Karuvath, ¶0028, a font size of the recommended items are adjusted in real time in accordance with the distance between the person and the display; Karuvath, ¶0029, attributes of a facial pattern of a person (such as gaze time, geographical location and facial expressions) are recorded and saved).
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of Karuvath with Yonaha and Yumiya because they are in the same field of endeavor. One skilled in the art would have been motivated to include adjustable character / font size as taught by Karuvath in the system of Yonaha and Yumiya in order to provide a means to provide a product recommendation to a person (Karuvath, ¶0019).
As per claim 19, Yonaha and Yumiya disclose the processing apparatus according to Claim 1, but do not explicitly disclose the following limitations as further recited however Karuvath discloses wherein the characteristic information includes a font used (Karuvath, ¶0027, retrieve recommended items based on an identification of the person. The recommended items are displayed at a display located along the footpath of the target person ... this item could be information about a product (e.g., an advertisement or other information that informs the person about the product); Karuvath, ¶0028, a font size of the recommended items are adjusted in real time in accordance with the distance between the person and the display; Karuvath, ¶0029, attributes of a facial pattern of a person (such as gaze time, geographical location and facial expressions) are recorded and saved).
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of Karuvath with Yonaha and Yumiya because they are in the same field of endeavor. One skilled in the art would have been motivated to include adjustable character / font size as taught by Karuvath in the system of Yonaha and Yumiya in order to provide a means to provide a product recommendation to a person (Karuvath, ¶0019).
Claim(s) 18 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yonaha, Japanese Patent Publication JP 2010268158A, hereinafter, “Yonaha”, in view of Yumiya H, Deguchi D, Kawanishi Y, Murase H. End-to-End Gaze Grounding of a Person Pictured from Behind. In VISIGRAPP (5: VISAPP) 2023 Feb 19-21 (pp. 898-905), hereinafter, “Yumiya” as applied to claim 1 above, and further in view of McQueen et al., U.S. Patent No. 9,911,398, hereinafter, “McQueen”.
As per claim 18, Yonaha and Yumiya disclose the processing apparatus according to Claim 1, but do not explicitly disclose the following limitations as further recited however McQueen discloses wherein the characteristic information includes a contrast difference between a background and a character (McQueen, column 6, lines 40-60, the optical/surface analysis component 412 may be configured to perform any combination of one or more of the following ... object recognition; product recognition; bar code detection and decoding; skeletal extraction and tracking; gaze detection; user attention tracking; body and body part detection; proximity detection; background subtraction ... distance determination; McQueen, column 9, lines 15-17, content may be presented in different sizes or at different levels of detail depending on user distance; McQueen, column 11, line 65 - column 12, 4, When the user 106 is at distance between the first distance D1 and a second distance D2, where D2 may be in the range of 5 feet as an example, a somewhat lower content density is selected and used for displaying the content 502. At this content density, the content 502 has fewer topic sections and may also have larger font sizes and icons so that they can be seen from the larger distances).
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of McQueen with Yonaha and Yumiya because they are in the same field of endeavor. One skilled in the art would have been motivated to include background difference as taught by McQueen in the system of Yonaha and Yumiya in order to provide a means to clearly provide information to a user (McQueen, column 1, lines 56-60).
As per claim 20, Yonaha and Yumiya disclose the processing apparatus according to Claim 1, but do not explicitly disclose the following limitations as further recited however McQueen discloses wherein the characteristic information include a shape of the target object (McQueen, column 6, lines 10-12, identify products and other articles by optical inspection and/or shape analysis; McQueen, column 6, lines 25-28, analyze the 3D or depth data to identify objects and to determine their shapes, distances, and positions, as well as to identify objects based on their shapes; McQueen, column 6, lines 40-60, the optical/surface analysis component 412 may be configured to perform any combination of one or more of the following ... object recognition; product recognition; bar code detection and decoding; skeletal extraction and tracking; gaze detection; user attention tracking; body and body part detection; proximity detection; background subtraction ... distance determination).
It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to combine the teachings of McQueen with Yonaha and Yumiya because they are in the same field of endeavor. One skilled in the art would have been motivated to include shape determination as taught by McQueen in the system of Yonaha and Yumiya in order to provide a means to clearly provide information to a user (McQueen, column 1, lines 56-60).
Conclusion
Applicants’ amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TRACY MANGIALASCHI whose telephone number is (571)270-5189. The examiner can normally be reached M-F, 9:30AM TO 6:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vu Le can be reached at (571) 272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TRACY MANGIALASCHI/Primary Examiner, Art Unit 2668
Prosecution Insights