Detailed Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d).
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 08/30/2025 has been entered.
Response to Amendments
Regarding applicant’s amendments to the claims filed on 08/30/2025, the amendments have been acknowledged, accepted, and entered. Previously claims 1-5, 11-19, 23 and 24 were pending; claims 6 – 10, 17, and 20-22 have been cancelled. Claims 1 and 18 have been amended, and currently claims 1 – 5, 11 – 16, 18, 19 and 23 – 24 are still pending.
Response to Arguments
Regarding applicant’s 103 prior art rejection arguments, filed on 08/30/2025, with respect to independent claims claim 1 and 18, the arguments have been fully considered but they are not persuasive. The applicant argues that in claim 1 and 18, Price and Hosono do not teach or suggest “determine other classes not defined by the temporal directionality, the spatial directionality, the temporal counterpart, and the spatial counterpart as negative classes, flip the behavior data to add the negative class that the spatial directionality exists and the spatial counterpart does not exist, and play the behavior data backwards to add the negative class based on that the temporal directionality exists and the temporal counterpart does not exist.” However, Price teaches, in Section 3, Label-altering video transforms, Sub-Section 2, Invariant, Sub-Section 2, Equivariant, and Sub Section 3, Novel Generating and equations 4-7 and table 1, of the novel generating category, representing the negative class. The novel generating category is given to a class that undergoes a transform such as time reversal and horizontal flipping, and the transformed outcome no longer belongs to any known class. The novel generating / negative class is added when the behavior is flipped (horizontal flipping) and there is spatial directionality but no spatial counterpart class, and when the behavior is played backwards (time reversal) and temporal directionality exists but no temporal counterpart; as no spatial or temporal directionality would lead to an invariant class, and spatial and temporal directionality with respective counterparts would lead to equivariant class; the novel generating represents the negative class wherein spatial or temporal directionality exists with no respective counterpart.
Therefore, using the explanations that have been provided above by the examiner, the claim rejections for claims 1 and 18 are maintained. Furthermore, the previously filed dependent claims dependent on claims 1 and 18 respectively are still considered obvious as the rejection of the independent claims in maintained.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 2, 11-16, 18 and 23-24 are rejected under 35 U.S.C. 103 as being unpatentable over Price; Will et al.(Retro-Actions: Learning ‘Close’ by Time-Reversing ‘Open’ Videos; hereinafter simply referred to as Price) in view of Hosono; Takashi et al. (US 20220277592 A1; hereinafter simply referred to as Hosono)
Regarding independent claim 1, Price teaches
A behavior data augmenting, learning and recognition process (See, Section 3, sub section “Introducing LATs”, ¶ 1) comprising:
Classify class of behavior data by a behavior of an object (See section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7 wherein the class of behavior data by a behavior of an object is classified as Invariant when the label is maintained after transformation; Equivariant, when the label is changed after transformation and there is a counterpart of the behavior of the object; and Novel-Generating, when a new class is formed after transformation)
determine a spatiotemporal characteristic for each class of the behavior data by a behavior of the object (The spatiotemporal characteristics for each class of behavior data are defined as: Invariant, where the label is maintained after transformation; Equivariant, where the label is changed after transformation and there is a counterpart of the behavior; and Novel-Generating, where a new class is formed after transformation, See section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7) (See section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, “Invariant classes, Yi : classes whose examples maintain their label after transformation” … “Equivariant, Ye: classes whose examples change label after transformation … We thus define Ty(y) = y 0 , referring to (y, y0 ) as a pair of equivariant classes where y 0 is the counterpart of y and vice versa” … “3. Novel-generating, Yn: these include classes whose transformed examples no longer belong to any of the dataset’s classes”);
augment the behavior data (The behavior data is augmented using Label-altering transforms (LATs) including horizontal-flipping, and time-reversal, See Section 3.2 Applications of class transforms, Subsection Data augmentation, Section 3.3 LAT examples, Subsection Transform 1: horizontal-flipping, and Subsection Transform 2: time-reversal) (See Section 3.2 Applications of class transforms, Subsection Data augmentation, Section 3.3 LAT examples, Subsection Transform 1: horizontal-flipping, and Subsection Transform 2: time-reversal, “We propose using LATs for augmenting both invariant and equivariant classes through target-conditional data augmentation” … “While in some video datasets, horizontal-flipping is a LPT, it is a LAT when the dataset includes classes with a defining uni-directional horizontal movement, e.g. ‘swipe right’ or ‘rotate clockwise” … “For example, time reversing an action such as ‘cover’ reverses the state change to produce an ‘uncover’ action”); and
perform learning to recognize the behavior of the object based on the augmented behavior data and a learning algorithm (The behavior of the object based on the augmented/transformed behavior data is recognized via the trained model f^ and the use of Zero-shot learning, where the behavior of the object is determined after augmentation as seen in Figure 2 and table 1, where the transforms result in different behaviors being recognized, See Section 3.1 Discovering Class Transforms ¶ 1, Section 4 Datasets and perception study, Subsection Class transforms ¶ 2, Section 3.1 Subsection Zero-shot learning and Figure 2, and Table 1 and , Section 3, sub section “Introducing LATs”, ¶ 1). (See Section 3.1 Discovering Class Transforms ¶ 1, Section 4 Datasets and perception study, Subsection Class transforms ¶2, and Table 1, and Figure 2 and Section 3, sub section “Introducing LATs, ¶ 1, “In order to automatically determine the class transform Ty, we propose a method based on the response of the trained model” … “As the table shows, time-reversal results in more equivariant classes than horizontal-flipping. We find 5 and 28 novel-generating reversible classes in Jester and Something-Something where the transformed label is not part of the label set” … “Given an oracle video labelling function f and a dataset with videos V and labels Y = {f(v)| v ∈ V }, we aim to learn the parameters of a model ˆf using the videos V and the supervision Y . We define a video transform T as an operation that takes a video v ∈ V and transforms it into another video vˆ = T(v) that is a valid input to the trainable model ˆf.”)
determine whether temporal directionality exists for each class of the behavior of the object, whether spatial directionality exists, a temporal counterpart when the video data is played backwards, and a spatial counterpart when the video data is flipped left and right. (The classes are placed into three different categories being invariant, equivariant and novel-generating after a transform has occurred. The transforms are horizontal flipping which indicates video data being flipped left and right, and time-reversal which indicates a video being played backwards. The category of class being equivariant indicates that there is a counterpart of the video data after a transform has been performed, while the invariant class indicates there is no counterpart as the label for the behavior of the class is unchanged after the transform; the category decided for each class after transform dictates whether or not spatial or temporal directionality exists and whether there is a temporal or spatial counterpart after the transformation, the transformations being horizontal flipping or time-reversal, See Section 3.3 LAT examples, Subsection Transform 1: horizontal-flipping, and Subsection Transform 2: time-reversal, section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1) (See Section 3.3 LAT examples, Subsection Transform 1: horizontal-flipping, and Subsection Transform 2: time-reversal, section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1, “Invariant classes, Yi : classes whose examples maintain their label after transformation” … “Equivariant, Ye: classes whose examples change label after transformation … We thus define Ty(y) = y 0 , referring to (y, y0 ) as a pair of equivariant classes where y 0 is the counterpart of y and vice versa” … “3. Novel-generating, Yn: these include classes whose transformed examples no longer belong to any of the dataset’s classes” … ““While in some video datasets, horizontal-flipping is a LPT, it is a LAT when the dataset includes classes with a defining uni-directional horizontal movement, e.g. ‘swipe right’ or ‘rotate clockwise” … “For example, time reversing an action such as ‘cover’ reverses the state change to produce an ‘uncover’ action”);
determining that the spatial directionality exists when the behavior of the object changes when the video data is flipped left to right. (Horizontal flipping transformations are performed on the video datasets and placed into three categories being invariant, equivariant, and novel-generating, if the class changes its behavior label after the horizontal flipping transformation and is placed in the category of equivariant then spatial directionality exists as the behavior changes after the horizontal flipping transformation occurs which can be seen in figure 2 and table 1, See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1) (See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1, “Invariant classes, Yi : classes whose examples maintain their label after transformation” … “Equivariant, Ye: classes whose examples change label after transformation … We thus define Ty(y) = y 0 , referring to (y, y0 ) as a pair of equivariant classes where y 0 is the counterpart of y and vice versa” … “3. Novel-generating, Yn: these include classes whose transformed examples no longer belong to any of the dataset’s classes”)
determining a different class as the spatial counterpart when the spatial directionality exists and the video data is treated as the different class when flipped left and right. (Horizontal flipping transformations are performed on the video datasets and placed into three categories being invariant, equivariant, and novel-generating, if the class changes its behavior label after the horizontal flipping transformation and is placed in the category of equivariant then spatial directionality exists as the behavior changes after the horizontal flipping transformation occurs and a new counterpart class “y 0” is defined which can be seen in figure 2 and table 1, See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1) (See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1, “Invariant classes, Yi : classes whose examples maintain their label after transformation” … “Equivariant, Ye: classes whose examples change label after transformation … We thus define Ty(y) = y 0 , referring to (y, y0 ) as a pair of equivariant classes where y 0 is the counterpart of y and vice versa” … “3. Novel-generating, Yn: these include classes whose transformed examples no longer belong to any of the dataset’s classes”)
determining that the temporal directionality exists when the behavior of the object is the same only in forward playback of the video data. (Time reversal transformations are performed on the video datasets and placed into three categories being invariant, equivariant, and novel-generating, if the class changes its behavior label after the time-reversal transformation and is placed in the category of equivariant then temporal directionality exists as the behavior is the same in a forward playback but not in a reverse playback which can be seen in figure 2 and table 1, See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1) (See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1, “Invariant classes, Yi : classes whose examples maintain their label after transformation” … “Equivariant, Ye: classes whose examples change label after transformation … We thus define Ty(y) = y 0 , referring to (y, y0 ) as a pair of equivariant classes where y 0 is the counterpart of y and vice versa” … “3. Novel-generating, Yn: these include classes whose transformed examples no longer belong to any of the dataset’s classes”)
determining a different class as the temporal counterpart when the temporal directionality exists and the video data is treated as the different class when played backwards. (Time reversal transformations are performed on the video datasets and placed into three categories being invariant, equivariant, and novel-generating, if the class changes its behavior label after the time-reversal transformation and is placed in the category of equivariant then temporal directionality exists as the behavior is the same in a forward playback but not in a reverse playback and a new counterpart class “y 0” is defined which can be seen in figure 2 and table 1, See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1) (See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1, “Invariant classes, Yi : classes whose examples maintain their label after transformation” … “Equivariant, Ye: classes whose examples change label after transformation … We thus define Ty(y) = y 0 , referring to (y, y0 ) as a pair of equivariant classes where y 0 is the counterpart of y and vice versa” … “3. Novel-generating, Yn: these include classes whose transformed examples no longer belong to any of the dataset’s classes”)
determine other classes not defined by the temporal directionality, the spatial directionality, the temporal counterpart, and the spatial counterpart as negative classes, flip the behavior data to add the negative class that the spatial directionality exists and the spatial counterpart does not exist, and play the behavior data backwards to add the negative class based on that the temporal directionality exists and the temporal counterpart does not exist. (See Price Section 3, Label-altering video transforms, Sub-Section 2, Invariant, Sub-Section 2, Equivariant, and Sub Section 3, Novel Generating and equations 4-7 and table 1, wherein the novel generating category, representing the negative class, is given to a class that undergoes a transform such as time reversal and horizontal flipping, and the transformed outcome no longer belongs to any known class. The novel generating / negative class is added when the behavior is flipped (horizontal flipping) and there is spatial directionality but no spatial counterpart class, and when the behavior is played backwards (time reversal) and temporal directionality exists but no temporal counterpart; as no spatial or temporal directionality would lead to an invariant class, and spatial and temporal directionality with respective counterparts would lead to equivariant class; thus the presence of temporal or spatial directionality lacking a respective spatial or temporal counterpart leads to a novel generating / negative class).
While it can be said that image recognition, as is performed by Price, necessarily requires the extraction of object regions from image/video data, and is necessarily performed using some form of processor implementing an algorithm, it is not explicitly stated. Therefore, Price does not teach of a behavior data augmenting apparatus comprising: a non-transitory memory storing algorithms and data; and a processor configured to execute the algorithms stored in the memory to: extract an object region from video data.
However, Hosono teaches of a behavior data augmenting apparatus (data augmenting apparatus: direction alignment unit, which performs behavior data augmentation in the form of rotations and inversions, See ¶ 8, 9, 44, 52) (See ¶ 8, “a direction alignment unit configured to perform at least one of rotation and inversion on the image based on an action direction of the desired subject in the image or an action direction of a subject other than the desired subject, so as to obtain an adjusted image”) comprising:
a non-transitory memory storing algorithms and data; and a processor configured to execute the algorithms stored in the memory to (non – transitory memory storing algorithms and data; and a processor configured to execute the algorithms stored in memory: The programs and data are stored in a non-transitory storage medium executed by a CPU (processor) to execute learning processing (including algorithms), See ¶ 126, 127, and 137) (See ¶ 126, 127, “The various types of processing that are executed by the CPU reading out and executing software (programs) in the above-described embodiments may be executed by various types of processors other than the CPU. … Also, the learning processing and action recognition processing may be executed by one of these various types of processors” … “The programs may be provided in a form of being stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory.”);
extract an object region from video data (The object region is extracted from video data via the use of the object detection unit 20, which estimates the object region for each frame image of a video, See ¶ 42, 47, 66, 82, 84, and Figure 3), (See ¶ 42, “The object detection unit 20 estimates the type of the subject and an object region that represents this subject, for each of frame images of the input video.”)
As taught by Hosono extracting an object region allows for alignment to occur to obtain adjusted images. (The use of the object detection unit, which extracts an object region, allows for a process to occur where the direction alignment unit 24 uses the results of the object detection unit to make action directions of the subject aligned with the reference direction creating a proper adjusted image, See ¶ 52, and Figures 3 and 4) (“The direction alignment unit 24 performs at least one of rotation and inversion on the video so that action directions of the desired subject are aligned with the reference direction, based on the object detection result and the optical flow calculation result, thereby obtaining adjusted images.”). As both the teachings of Price and Hosono deal with the technical field of augmenting images; it would have been obvious to one of ordinary skill in the art before the effective filling date of the claimed invention to combine the teachings of Price with Hosono to use a non-transitory memory storing algorithms and data executed by a processor to implement a behavior data augmenting apparatus to extract an object region from video data in order to allow for alignment to create a proper adjusted image after augmentation.
Regarding dependent claim 2, Price in view of Hosono teaches;
The behavior data augmenting apparatus of claim 1(data augmenting apparatus: direction alignment unit, which performs behavior data augmentation in the form of rotations and inversions, See Hosono ¶ 8, 9, 44, 52) (See Hosono ¶ 8, “a direction alignment unit configured to perform at least one of rotation and inversion on the image based on an action direction of the desired subject in the image or an action direction of a subject other than the desired subject, so as to obtain an adjusted image”) wherein the processor is configured to execute the algorithms (The programs and data are stored in a non-transitory storage medium executed by a CPU (processor) to execute learning processing (including algorithms), See Hosono ¶ 126, 127, and 137) (See Hosono ¶ 126, 127, “The various types of processing that are executed by the CPU reading out and executing software (programs) in the above-described embodiments may be executed by various types of processors other than the CPU. … Also, the learning processing and action recognition processing may be executed by one of these various types of processors” … “The programs may be provided in a form of being stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory.”); extracting the object region for each frame of the video data by using an object detection algorithm. (The object detection unit is capable of detecting an object using any promising method, including an object detection algorithm, examples of promising methods are given via Reference Document 1 (“Mask R-CNN” by K. He et al.) and Reference document 2 (“Simple online and realtime tracking” by A. Bewley et al.) both containing object detection algorithms. See Hosono ¶ 47, 48, and 49) (See Hosono ¶ 47, 48, and 49, “The object detection unit 20 detects the type and position of a desired subject (for example, a person or an object operated by a person). Any promising method can be used as the object detection method. For example, an object detection method as disclosed by Reference Document 1 can be performed on each frame image to realize object detection. Also, by performing an object tracking method as disclosed in Reference Document 2 on an object detection result of the first frame, the type and position of an object from the second frames onwards may also be estimated.”)
Regarding dependent claim 11, Price in view of Hosono teaches:
The behavior data augmenting apparatus of claim 1 (data augmenting apparatus: direction alignment unit, which performs behavior data augmentation in the form of rotations and inversions, See Hosono ¶ 8, 9, 44, 52) (See Hosono ¶ 8, “a direction alignment unit configured to perform at least one of rotation and inversion on the image based on an action direction of the desired subject in the image or an action direction of a subject other than the desired subject, so as to obtain an adjusted image”) wherein the processor is configured to execute the algorithms (The programs and data are stored in a non-transitory storage medium executed by a CPU (processor) to execute learning processing (including algorithms), See Hosono ¶ 126, 127, and 137) (See Hosono ¶ 126, 127, “The various types of processing that are executed by the CPU reading out and executing software (programs) in the above-described embodiments may be executed by various types of processors other than the CPU. … Also, the learning processing and action recognition processing may be executed by one of these various types of processors” … “The programs may be provided in a form of being stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory.”); generating a new behavior as new second class data when the new behavior is detected when first class data having the temporal directionality is played backwards. (Time reversal transformations are performed on the video datasets and placed into three categories being invariant, equivariant, and novel-generating, if the class changes its behavior label after the time-reversal transformation, then temporal directionality exists as the behavior is the same in a forward playback but not in a reverse playback and is assigned the novel-generating category or equivariant category depending on if there is a counterpart, as a new label behavior for a class is created which can be seen in figure 2 and table 1, See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1) (See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1, “Invariant classes, Yi : classes whose examples maintain their label after transformation” … “Equivariant, Ye: classes whose examples change label after transformation … We thus define Ty(y) = y 0 , referring to (y, y0 ) as a pair of equivariant classes where y 0 is the counterpart of y and vice versa” … “3. Novel-generating, Yn: these include classes whose transformed examples no longer belong to any of the dataset’s classes”)
Regarding dependent claim 12, Price in view of Hosono teaches:
The behavior data augmenting apparatus of claim 1 (data augmenting apparatus: direction alignment unit, which performs behavior data augmentation in the form of rotations and inversions, See Hosono ¶ 8, 9, 44, 52) (See Hosono ¶ 8, “a direction alignment unit configured to perform at least one of rotation and inversion on the image based on an action direction of the desired subject in the image or an action direction of a subject other than the desired subject, so as to obtain an adjusted image”) wherein the processor is configured to execute the algorithms (The programs and data are stored in a non-transitory storage medium executed by a CPU (processor) to execute learning processing (including algorithms), See Hosono ¶ 126, 127, and 137) (See Hosono ¶ 126, 127, “The various types of processing that are executed by the CPU reading out and executing software (programs) in the above-described embodiments may be executed by various types of processors other than the CPU. … Also, the learning processing and action recognition processing may be executed by one of these various types of processors” … “The programs may be provided in a form of being stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory.”); generating a new behavior as new second class data when the new behavior is detected when first class data having the spatial directionality is flipped left and right. (Horizontal flipping transformations are performed on the video datasets and placed into three categories being invariant, equivariant, and novel-generating, if the class changes its behavior label after the horizontal flipping transformation, then spatial directionality exists as the behavior changes after the horizontal flipping transformation occurs and is assigned the novel-generating category or equivariant category depending on if there is a counterpart, as a new label behavior for a class is created which can be seen in figure 2 and table 1, See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1) (See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1, “Invariant classes, Yi : classes whose examples maintain their label after transformation” … “Equivariant, Ye: classes whose examples change label after transformation … We thus define Ty(y) = y 0 , referring to (y, y0 ) as a pair of equivariant classes where y 0 is the counterpart of y and vice versa” … “3. Novel-generating, Yn: these include classes whose transformed examples no longer belong to any of the dataset’s classes”)
Regarding dependent claim 13, Price in view of Hosono teaches:
The behavior data augmenting apparatus of claim 1 (data augmenting apparatus: direction alignment unit, which performs behavior data augmentation in the form of rotations and inversions, See Hosono ¶ 8, 9, 44, 52) (See Hosono ¶ 8, “a direction alignment unit configured to perform at least one of rotation and inversion on the image based on an action direction of the desired subject in the image or an action direction of a subject other than the desired subject, so as to obtain an adjusted image”) wherein the processor is configured to execute the algorithms (The programs and data are stored in a non-transitory storage medium executed by a CPU (processor) to execute learning processing (including algorithms), See Hosono ¶ 126, 127, and 137) (See Hosono ¶ 126, 127, “The various types of processing that are executed by the CPU reading out and executing software (programs) in the above-described embodiments may be executed by various types of processors other than the CPU. … Also, the learning processing and action recognition processing may be executed by one of these various types of processors” … “The programs may be provided in a form of being stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory.”); storing and augmenting first class data having no temporal directionality when a same behavior as that of the first class data is detected when the first class data is played backwards (Time reversal transformations/augmentations are performed on the video datasets and placed into three categories being invariant, equivariant, and novel-generating, if the class does not change its behavior label after the time-reversal transformation, then temporal directionality does not exists as the behavior is the same in a forward playback as in reverse playback and is assigned the invariant category which can be seen in figure 2 and table 1, the transformation data is stored in a variable, See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1) (See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1, “Invariant classes, Yi : classes whose examples maintain their label after transformation” … “Equivariant, Ye: classes whose examples change label after transformation … We thus define Ty(y) = y 0 , referring to (y, y0 ) as a pair of equivariant classes where y 0 is the counterpart of y and vice versa” … “3. Novel-generating, Yn: these include classes whose transformed examples no longer belong to any of the dataset’s classes”)
Regarding dependent claim 14, Price in view of Hosono teaches:
The behavior data augmenting apparatus of claim 1 (data augmenting apparatus: direction alignment unit, which performs behavior data augmentation in the form of rotations and inversions, See Hosono ¶ 8, 9, 44, 52) (See Hosono ¶ 8, “a direction alignment unit configured to perform at least one of rotation and inversion on the image based on an action direction of the desired subject in the image or an action direction of a subject other than the desired subject, so as to obtain an adjusted image”) wherein the processor is configured to execute the algorithms (The programs and data are stored in a non-transitory storage medium executed by a CPU (processor) to execute learning processing (including algorithms), See Hosono ¶ 126, 127, and 137) (See Hosono ¶ 126, 127, “The various types of processing that are executed by the CPU reading out and executing software (programs) in the above-described embodiments may be executed by various types of processors other than the CPU. … Also, the learning processing and action recognition processing may be executed by one of these various types of processors” … “The programs may be provided in a form of being stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory.”); storing and augmenting first class data having no spatial directionality when a same behavior as that of the first class data is detected when the first class data is flipped left and right (Horizontal flipping transformations/augmentations are performed on the video datasets and placed into three categories being invariant, equivariant, and novel-generating, if the class does not change its behavior label after the Horizontal flipping transformation, then spatial directionality does not exists as the behavior is the same after flipping left and right and is assigned the invariant category which can be seen in figure 2 and table 1, the transformation data is stored in a variable, See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1) (See Price section 3, Label-altering video transforms, Sub-Section 1, Invariant, Sub-Section 2, Equivariant, and Sub-Section 3, Novel-generating, and Equations 4-7, and figure 2 and Table 1, “Invariant classes, Yi : classes whose examples maintain their label after transformation” … “Equivariant, Ye: classes whose examples change label after transformation … We thus define Ty(y) = y 0 , referring to (y, y0 ) as a pair of equivariant classes where y 0 is the counterpart of y and vice versa” … “3. Novel-generating, Yn: these include classes whose transformed examples no longer belong to any of the dataset’s classes”)
Regarding dependent claim 15, Price in view of Hosono teaches:
The behavior data augmenting apparatus of claim 1 (data augmenting apparatus: direction alignment unit, which performs behavior data augmentation in the form of rotations and inversions, See Hosono ¶ 8, 9, 44, 52) (See Hosono ¶ 8, “a direction alignment unit configured to perform at least one of rotation and inversion on the image based on an action direction of the desired subject in the image or an action direction of a subject other than the desired subject, so as to obtain an adjusted image”) wherein the processor is configured to execute the algorithms (The programs and data are stored in a non-transitory storage medium executed by a CPU (processor) to execute learning processing (including algorithms), See Hosono ¶ 126, 127, and 137) (See Hosono ¶ 126, 127, “The various types of processing that are executed by the CPU reading out and executing software (programs) in the above-described embodiments may be executed by various types of processors other than the CPU. … Also, the learning processing and action recognition processing may be executed by one of these various types of processors” … “The programs may be provided in a form of being stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory.”); augmenting same class data by randomly sampling a plurality of templates in terms of time in a learning phase. (TRNs (Temporal Relational Network) are used in the learning/training phase which constitute splitting the video, consisting of same class data, into a number of segments/templates which are then randomly sampled; a video is inherently a representation of space and time so the randomly selected segment/template that is sampled is in terms of time, See Price Section 5 Experiments and Results Subsection 1 Implementation details, ¶ 1 and 2 ) (See Price Section 5 Experiments and Results Subsection 1 Implementation details, ¶ 1 and 2 “We employ a Temporal Relational Network (TRN) [30] with a batch-normalised Inception (BNInception) backbone [15] trained on RGB video due to its temporal-sensitivity, computational efficiency through sparse sampling, and high performance on benchmark datasets (including those we test on). In TRNs, the input video is split into n segments from which a frame is randomly sampled.” … “In all experiments, we train our networks for 100 epochs with an initial learning rate of 1 × 10−3 divided by 10 at epochs 40 and 80. We use a batch size of 80 for Jester and 128 for Something-Something training on 4 GPUs. All other parameters follow the default values from the TRN GitHub codebase”)
Regarding dependent claim 16, Price in view of Hosono teaches:
The behavior data augmenting apparatus of claim 1 (data augmenting apparatus: direction alignment unit, which performs behavior data augmentation in the form of rotations and inversions, See Hosono ¶ 8, 9, 44, 52) (See Hosono ¶ 8, “a direction alignment unit configured to perform at least one of rotation and inversion on the image based on an action direction of the desired subject in the image or an action direction of a subject other than the desired subject, so as to obtain an adjusted image”) wherein the processor is configured to execute the algorithms (The programs and data are stored in a non-transitory storage medium executed by a CPU (processor) to execute learning processing (including algorithms), See Hosono ¶ 126, 127, and 137) (See Hosono ¶ 126, 127, “The various types of processing that are executed by the CPU reading out and executing software (programs) in the above-described embodiments may be executed by various types of processors other than the CPU. … Also, the learning processing and action recognition processing may be executed by one of these various types of processors” … “The programs may be provided in a form of being stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory.”); augmenting same class data by randomly sampling a plurality of templates in terms of space in a learning phase. (TRNs (Temporal Relational Network) are used in the learning/training phase which constitute splitting the video, consisting of same class data, into a number of segments/templates which are then randomly sampled; a video is inherently a representation of space and time so the randomly selected segment/template that is sampled is in terms of space, See Price Section 5 Experiments and Results Subsection 1 Implementation details, ¶ 1 and 2 ) (See Price Section 5 Experiments and Results Subsection 1 Implementation details, ¶ 1 and 2 “We employ a Temporal Relational Network (TRN) [30] with a batch-normalised Inception (BNInception) backbone [15] trained on RGB video due to its temporal-sensitivity, computational efficiency through sparse sampling, and high performance on benchmark datasets (including those we test on). In TRNs, the input video is split into n segments from which a frame is randomly sampled.” … “In all experiments, we train our networks for 100 epochs with an initial learning rate of 1 × 10−3 divided by 10 at epochs 40 and 80. We use a batch size of 80 for Jester and 128 for Something-Something training on 4 GPUs. All other parameters follow the default values from the TRN GitHub codebase”)
Regarding independent claim 18, claim 18 is a method claim corresponding to claim 1. Please see the discussion of 1 above.
Regarding dependent claim 23, claim 23 is a method claim corresponding to claim 11. Please see the discussion of 11 above.
Regarding dependent claim 24, claim 24 is a method claim corresponding to claim 12. Please see the discussion of 12 above.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Price; Will et al.(Retro-Actions: Learning ‘Close’ by Time-Reversing ‘Open’ Videos; hereinafter simply referred to as Price) in view of Hosono; Takashi et al. (US 20220277592 A1; hereinafter simply referred to as Hosono) further in view of Ramola; Gaurav (US 20220230331 A1; hereinafter simply referred to as Ramola)
Regarding dependent claim 3, Price in view of Hosono teaches:
The behavior data augmenting apparatus of claim 1 (data augmenting apparatus: direction alignment unit, which performs behavior data augmentation in the form of rotations and inversions, See Hosono ¶ 8, 9, 44, 52) (See Hosono ¶ 8, “a direction alignment unit configured to perform at least one of rotation and inversion on the image based on an action direction of the desired subject in the image or an action direction of a subject other than the desired subject, so as to obtain an adjusted image”) wherein the processor is configured to execute the algorithms (The programs and data are stored in a non-transitory storage medium executed by a CPU (processor) to execute learning processing (including algorithms), See Hosono ¶ 126, 127, and 137) (See Hosono ¶ 126, 127, “The various types of processing that are executed by the CPU reading out and executing software (programs) in the above-described embodiments may be executed by various types of processors other than the CPU. … Also, the learning processing and action recognition processing may be executed by one of these various types of processors” … “The programs may be provided in a form of being stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory.”);
Price in view of Hosono does not teach recognizing the object based on an entire screen of a frame without detecting the object region for each frame of the video data, however;
Ramola teaches recognizing the object based on an entire screen of a frame without detecting the object region for each frame of the video data (The video motion controller 140, is able to detect an object in each image frame of a video by looking at the entire frame without first setting an object region. See ¶ 47) (See ¶ 47 “The video motion controller (140) may include various control circuitry and is configured to detect a first object and a second object in an image frame of the video.”)
As taught by Ramola using the entire screen of the frame without specifying an object region allows for more than one object to be identified (The use of a video motion controller allows the detection of multiple objects in the frame not just limited to finding a single object in an object region. See Ramola ¶ 21, 47, 68) (See Ramola ¶ 68 “The operations (S402-S406) may be performed by the video motion controller (140). At S402, the method includes detecting the first object and the second object in the image frame of the video.”) As both the teachings of Price in view of Hosono and Ramola deal with the technical field of detecting objects in a video, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed in invention to combine the teachings of Price in view of Hosono and Ramola to recognize the object based on an entire screen of the frame without detecting the object region for each frame of the video data in order to be able to detect multiple objects in a video frame without being constrained by an object region.
Claims 4, 5, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Price; Will et al.(Retro-Actions: Learning ‘Close’ by Time-Reversing ‘Open’ Videos; hereinafter simply referred to as Price) in view of Hosono; Takashi et al. (US 20220277592 A1; hereinafter simply referred to as Hosono) further in view of Gum; Arnold (KR20140148448A; hereinafter simply referred to as Gum; translated via Google Patents)
Regarding dependent claim 4, Price in view of Hosono teaches:
The behavior data augmenting apparatus of claim 1 (data augmenting apparatus: direction alignment unit, which performs behavior data augmentation in the form of rotations and inversions, See Hosono ¶ 8, 9, 44, 52) (See Hosono ¶ 8, “a direction alignment unit configured to perform at least one of rotation and inversion on the image based on an action direction of the desired subject in the image or an action direction of a subject other than the desired subject, so as to obtain an adjusted image”) wherein the processor is configured to execute the algorithms (The programs and data are stored in a non-transitory storage medium executed by a CPU (processor) to execute learning processing (including algorithms), See Hosono ¶ 126, 127, and 137) (See Hosono ¶ 126, 127, “The various types of processing that are executed by the CPU reading out and executing software (programs) in the above-described embodiments may be executed by various types of processors other than the CPU. … Also, the learning processing and action recognition processing may be executed by one of these various types of processors” … “The programs may be provided in a form of being stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory.”);
Price in view of Hosono does not teach selecting the object having a highest reliability when at least two objects exist in one frame, however;
Gum teaches selecting the object having a highest reliability when at least two objects exist in one frame. (The selection of an object when there are multiple objects in a frame is decided by which object is positioned closer to the center of the image, representing reliability, See Gum Page 7, ¶ 6) (See Gum Page 7, ¶ 6, “In addition to object motion, some implementations may select one or more objects to focus on based at least in part on other object features. For example, the selection of an object may also be based on the location of the object in the image. For example, objects positioned closer to the center of the image may be prioritized higher than objects located closer to the edge of the image.”)
As taught by Gum using reliability as an object feature allows for autofocus implementations to be adjusted allowing for clear images to be obtained, (The use of reliability as a feature to determine the autofocus on an object in an image allows for high quality images to be captured in various environments, See Gum Page 2, Section Description ¶ 3) (See Gum Page 2, Section Description ¶ 3 Page 7 ¶ 7,“The focus priority may be increased or decreased based on the color of the object or the relative position of the object in the image. In these implementations, after the focus priority of each object is adjusted based on these and other object features, the object with the highest priority may be selected for focus.”) As both the teachings of Price in view of Hosono and Gum deal with the technical field of detecting an object in a video, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed inv