DETAILED ACTION
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be /negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over HASKIN (WO 2024042508 A1), and in view of BRUNNER (WO 2023115016 A1), and further in view of LIN (US 20220014807 A1).
Re Claim 1, HASKIN discloses A system comprising:
a processor; and
a memory communicably coupled to the processor, wherein the memory comprises processor-executable instructions which, when executed by the processor (see HASKIN: e.g., Fig. 1, and Fig. 8; --An image processor block 13 receives the analog signal from the image sensor 12. The Analog Front End (AFE) in the block 13 filters, amplifies, and digitizes the signal, using an analog-to-digital (A/D) converter. The AFE further provides Correlated Double Sampling (CDS) and provides a gain control to accommodate varying illumination conditions. In the case of a CCD-based sensor 12, a CCD AFE (Analog Front End) component may be used between the digital image processor 13 and the sensor 12. Such an AFE may be based on VSP2560 ‘CCD Analog Front End for Digital Cameras’ available from Texas Instruments Incorporated of Dallas, Texas, U.S.A. The block 13 further contains a digital image processor, which receives the digital data from the AFE, and processes this digital representation of the image to handle various industry standards, and executes various computations and algorithms. Preferably, additional image enhancements may be performed by the block 13 such as generating greater pixel density or adjusting color balance, contrast, and luminance. Further, the block 13 may perform other data management functions and processing on the raw digital image data. Commonly, the timing relationship of the vertical / horizontal reference signals and the pixel clock are also handled in this block. Digital Media System-on-Chip device TMS320DM357 available from Texas Instruments Incorporated of Dallas, Texas, U.S.A. is an example of a device implementing in a single chip (and associated circuitry) part or all of the image processor 13, part or all of a video compressor 14 and part or all of a transceiver 15.--, in Description section, lines 10-35, page 2) cause the processor to:
obtain a set of multimedia data feeds from a plurality of image capturing devices, wherein the set of multimedia data feeds correspond to a real-time event, and wherein the set of multimedia data feeds correspond to a time-series data captured at a plurality of time intervals and a multi-resolution data captured from the plurality of image capturing devices (see HASKIN: e.g., -- Streaming media is multimedia that is constantly received by and presented to an end-user while being delivered by a provider. … Live streaming refers to content delivered live over the Internet, and requires a form of source media (e.g. a video camera, an audio interface, screen capture software), an encoder to digitize the content, a media publisher, and a content delivery network to distribute and deliver the content. Streaming content may be according to, compatible with, or based on, IETF RFC 2550 entitled: “RTP: A Transport Protocol for Real-Time Applications”, IETF RFC 4587 entitled: “RTP Payload Format for H.261 Video Streams”, or IETF RFC 2326 entitled: “Real Time Streaming Protocol (RTSP)”--, in lines 10-21, page 43; also see: -- The detection mechanism is typically an electrical circuit including a re-triggerable multivibrator or another functionally similar component. Image. A digital image is a numeric representation (normally binary) of a two- dimensional image. Depending on whether the image resolution is fixed, it may be of a vector or raster type. Raster images have a finite set of digital values, called picture elements or pixels. The digital image contains a fixed number of rows and columns of pixels, which are the smallest individual element in an image, holding quantized values that represent the brightness of a given color at any specific point. Typically, the pixels are stored in computer memory as a raster image or raster map, a two-dimensional array of small integers, where these values are commonly transmitted or stored in a compressed form. The raster images can be created by a variety of input devices and techniques, such as digital cameras, scanners, coordinate-measuring machines, seismographic profiling, airborne radar, and more.--, in lines 1-11, page 11; and, -- Video content analysis transforms the audio and image stream into a set of semantically meaningful representations. The ultimate goal is to extract structural and semantic content automatically, without any human intervention, at least for limited types of video domains. Algorithms to perform content analysis include those for detecting objects in video, recognizing specific objects, persons, locations, detecting dynamic events in video, associating keywords with image regions or motion. VCA is used in a wide range of domains including entertainment, health-care, retail, automotive, transport, home automation, flame and smoke detection, safety and security. The algorithms can be implemented as software on general purpose machines, or as hardware in specialized video processing units. Many different functionalities can be implemented in VCA. Video Motion Detection is one of the simpler forms where motion is detected with regard to a fixed background scene. More advanced functionalities include video tracking and egomotion estimation. Based on the internal representation that VCA generates in the machine, it is possible to build other functionalities, such as identification, behavior analysis or other forms of situation awareness. VCA typically relies on good input video, so it is commonly combined with video enhancement technologies such as video denoising, image stabilization, unsharp masking and super- resolution--, in line 27, page 31 through line 15, page 32; also see: --YOLO is implemented as a CNN…. The YOLO approach to object detection describing frame object detection as a regression problem to spatially separated bounding boxes and associated class probabilities is described in an article authored by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi, published 9 May 2016 and entitled: “You Only Look Once: Unified, Real-Time Object Detection”, which is incorporated in its entirety for all purposes as if fully set forth herein. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance. The base YOLO model processes images in real- time at 45 frames per second while a smaller version of the network, Fast YOLO, processes an astounding 155 frames per second while still achieving double the mAP of other real-time detectors.--, in lines 10-32, page 60);
process the obtained set of multimedia data feeds using a plurality of model hyperparameters, wherein the plurality of hyperparameters comprise a frame rate, a domain specific semantic compression, a multimedia segment length, and overlap between multimedia segments (see HASKIN: e.g., -- and a video stream may be compressed using a video codec such as H.264 or VP8. Encoded audio and video streams may be assembled in a container bitstream such as MP4, FLV, WebM, ASF or ISMA. The bitstream is typically delivered from a streaming server to a streaming client using a transport protocol, such as MMS or RTP.--, in lines 26-31, page 43; and, -- Currently common HD modes are defined as 720p, 1080i, 1080p, and 1440p. Video mode 720p relates to frame size of 1,280×720 (W x H) pixels, 921,600 pixels per image, progressive scanning, and frame rates of 23.976, 24, 25, 29.97, 30, 50, 59.94, 60, or 72 Hz. Video mode 1080i relates to frame size of 1,920×1,080 (W x H) pixels, 2,073,600 pixels per image, interlaced scanning, and frame rates of 25 (50 fields/s), 29.97 (59.94 fields/s), or 30 (60 fields/s) Hz. Video mode 1080p relates to frame size of 1,920×1,080 (W x H) pixels, 2,073,600 pixels per image, progressive scanning, and frame rates of 24 (23.976), 25, 30 (29.97), 50, or 60 (59.94) Hz. Similarly, video mode 1440p relates to frame size of 2,560×1,440 (W x H) pixels, 3,686,400 pixels per image, progressive scanning, and frame rates of 24 (23.976), 25, 30 (29.97), 50, or 60 (59.94) Hz.--, in lines 1627, page 30; and,
--As for object detection accuracy, it achieves a mean Average Precision (mAP) of 64.16% for PASCAL VOC 2007 dataset that is only 2.63% lower than the mAP of the same YOLO network with full precision. R-CNN. Regions with CNN features (R-CNN) family is a family of machine learning models used to bypass the problem of selecting a huge number of regions. The R-CNN uses selective search to extract just 2000 regions from the image, referred to as region proposals. Then, instead of trying to classify a huge number of regions, only 2000 regions are handled. These 2000 region proposals are generated using a selective search algorithm, that includes Generating initial sub-segmentation for generating many candidate regions, using greedy algorithm to recursively combine similar regions into larger ones, and using the generated regions to produce the final candidate region proposals. These 2000 candidate region proposals are warped into a square and fed into a convolutional neural network that produces a 4096- dimensional feature vector as output.—, in lines 15-28, page 62; and,
--As in the original R-CNN, the Fast R-CNN uses Selective Search to generate its region proposals. While Fast R-CNN used Selective Search to generate ROIs, Faster R-CNN integrates the ROI generation into the neural network itself. Mask R-CNN adds instance segmentation, and also replaced ROIPooling with a new method called ROIAlign, which can represent fractions of a pixel, and Mesh R-CNN adds the ability to generate a 3D mesh from a 2D image. R-CNN and Fast R-CNN are primarily image classifier networks which are used for object detection by using Region Proposal method to generate potential bounding boxes in an image, run the classifier on these boxes, and after classification, perform post processing to tighten the boundaries of the bounding boxes and remove duplicates. Regions with CNN features (R-CNN) that combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost--, in lines 15-28, page 63;
-- A Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals, is described in an article authored by Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, published 2015, entitled: “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal networks”, which is incorporated in its entirety for all purposes as if fully set forth herein. State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. An RPN is a fully- convolutional network that simultaneously predicts object bounds and objectness scores at each position. RPNs are trained end-to-end to generate high quality region proposals, which are used by Fast R-CNN for detection. With a simple alternating optimization, RPN and Fast R-CNN can be trained to share convolutional features. For the very deep VGG-16 model, a described detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state- of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image. Code is available at https://github.com/ShaoqingRen/faster_rcnn. RetinaNet. RetinaNet is one of the one-stage object detection models that has proven to work well with dense and small-scale objects, that has become a popular object detection model to be used with aerial and satellite imagery. RetinaNet has been formed by making two improvements over existing single stage object detection models - Feature Pyramid Networks (FPN) and Focal Loss. Traditionally, in computer vision, featurized image pyramids have been used to detect objects with varying scales in an image. Featurized image pyramids are feature pyramids built upon image pyramids, where an image is subsampled into lower resolution and smaller size images (thus, forming a pyramid). Hand-engineered features are then extracted from each layer in the pyramid to detect the objects, which makes the pyramid scale-invariant.--, in line 31, page 64 through line 35, page 65; and, -- applying these mobile models to object detection in a novel framework referred to as SSDLite, and further demonstrates how to build mobile semantic segmentation models through a reduced form of DeepLabv3 (referred to as Mobile DeepLabv3), is based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers. The intermediate expansion layer uses lightweight depth-wise convolutions to filter features as a source of non-linearity. The scheme allows for decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis.--, in line 10, page 70 through line 8, page 71);
HASKIN however does not explicitly disclose sequence the processed set of multimedia data feeds in a predetermined order based on the time-scries data and the multi-resolution data;
BRUNNER discloses sequence the processed set of multimedia data feeds in a predetermined order based on the time-scries data and the multi-resolution data (see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, -- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; and, -- The use of two optical cameras can provide depth or 3D information. The particular configuration of sensors can be adapted for optimal capture of information regarding an object (e.g., an animal). Such information can include an instant assessment (namely features, variables, or primitives) of the object’s state or a sequence of such assessments in a time series. [0177] As described above, FIGS.3A-3B illustrates a system 310 for analyzing movements of an object 315, in accordance with one or more embodiments of the present disclosure. System 310 may include a control computer 50 communicating 335 with a computer- controlled enclosure 320 to which a plurality of sensors and/or actuators may be coupled thereon to execute an experiment on object 315, such as a rodent. The rodent may be administered, for example, with a compound in accordance with an experimental plan and observed with the plurality of sensors such that the system 310 may determine the behavioral and/or physiological effects of the administered compound on the object over a predetermined time period (e.g., the experimental session).--, in [0176]-[0177]);
HASKIN and BRUNNER are combinable as they are in the same field of endeavor: multimedia and video image data processing. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify HASKIN’s system using BRUNNER’s teachings by including sequence the processed set of multimedia data feeds in a predetermined order based on the time-scries data and the multi-resolution data to HASKIN’s multimedia data for object detection and analysis in order to observed with the plurality of sensors such that the system 310 may determine the behavioral and/or physiological effects of the administered compound on the object over a predetermined time period (see BRUNNER: e.g. in [0117]-[0119], [0144]-[0145], and [0176]-[0177]),
although HASKIN as modified by BRUNNER disclose receiving user input (see HASKIN: e.g., -- Similarly, the receiving path comprises a received channel in the RF spectrum, down-converted to the baseband (IF) wherein the data is then extracted. In order to support multiple devices and use a permanent solution, a Wireless Access Point (WAP) is typically used… Authentication in general, and a WAP authentication in particular, is used as the basis for authorization, which determines whether a privilege may be granted to a particular user or process, privacy, which keeps information from becoming known to non-participants, and non-repudiation, which is the inability to deny having done something that was authorized to be done based on the authentication. An authentication in general, and a WAP authentication in particular, may use an authentication server that provides a network service that applications may use to authenticate the credentials, usually account names and passwords of their users. When a client submits a valid set of credentials, it receives a cryptographic ticket that it can subsequently be used to access various services. Authentication algorithms include passwords, Kerberos, and public key encryption. Prior art technologies for data networking may be based on single carrier modulation techniques, such as AM (Amplitude Modulation), FM (Frequency Modulation), and PM (Phase Modulation), as well as bit encoding techniques such as QAM (Quadrature Amplitude Modulation) and QPSK (Quadrature Phase Shift Keying).--, in line 32, page 23 through line 14, page 24);
HASKIN as modified by BRUNNER however still do not explicitly disclose obtain at least one input prompt corresponding to the set of multimedia data feeds from at least one input source;
LIN discloses obtain at least one input prompt corresponding to the set of multimedia data feeds from at least one input source (see LIN: e.g., -- The length information can be obtained from the user. For example, it may send a prompt for asking the user to generate a long captioning information or a short captioning information, the user can give corresponding instructions according to the prompt. The length information can also be obtained by analyzing the video. When the video is a real-time captured video, it may determine the current application scenario by analyzing the video, and different length information can be determined for different application scenarios.--, in [0155], and, -- When the captioning information is generated by performing encoding, it may also generate a text caption of the video that better meets the requirements of the user by acquiring information about the captioning information that the user expects to generate. For example, during the user driving, it may collect real-time video in front of the user's line of sight and analyze the video, so as to give the user a corresponding reminder by analyzing the generated captioning information of the video or play the captioning information to the user when the user needs to be prompted, such as when there is a potential danger ahead.--, in [0366]);
HASKIN (as modified by BRUNNER) and LIN are combinable as they are in the same field of endeavor: multimedia and video image data processing. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify HASKIN (as modified by BRUNNER)’s system using LIN’s teachings by including obtain at least one input prompt corresponding to the set of multimedia data feeds from at least one input source to HASKIN (as modified by BRUNNER)’s user’s input in order to obtain information from the user, and generate a text caption of the video that better meets the requirements of the user (see BRUNNER: e.g. in [0155], and [0366]),
HASKIN as modified by BRUNNER and LIN further disclose generate an output representation of the real-time event by encoding the sequenced set of multimedia data feeds and the obtained at least one input prompt using a trained vision encoder model, wherein the generated output representation corresponds to a multi- resolution summary image of the real-time event at a time instance (see BRUNNER: e.g., --Some aspects of social interaction may occur when two subjects are in close proximity, creating occlusion from view from any unique camera setting. Quick movements of the nose may indicate an exploration of the social counterpart. Behavioral platforms consistent with disclosed embodiments can support camera mounting, resolution, and sampling rates that can enable automatic extraction of features corresponding to minute aspects of social behavior. In contrast, data acquired by conventional platforms may lack sufficient detail for automatic extraction and therefore may require manual extraction of such features. In addition, the approach to modelling the temporal aspects of all measured features may capture the complex temporal dynamics of such a social encounter. [0244] Consistent with disclosed embodiments, an analysis considering the temporal aspect of individual behavior and interactions between subjects may be more accurate and/or may provide more information than an analysis that summarizes such behavior or interactions over a period of time, such as the duration of a social encounter. For example, a temporal analysis using autocorrelation and lag analysis may indicate how the autocorrelation structure of natural behavior, or higher-order features of a subject change (e.g., changes in motifs, states, domains, or the like) as a result of the simple presence of a social counterpart.--, in [0243]; and, -- The segmentation and identification information for each video frame can be fundamental data to calculate the gait features. Given that the video image data is a 2D array, it can be summarized and converted into a tabular data format for further machine learning or statistical analysis steps. Tabulated data for each frame with the detected body part can be used as the source of information. For example, the center of the area can give the coordinate of the body or detected paws. The area and perimeter of the body area can provide shape information.--, in [0323]-[0324]; and, -- with regard to gait features summary, the at least one processor 360 may provide a summary by summarizing the gait features over multiple cycles. Features can be averaged separately for the different behaviors (all, other, walk), directions (CCW, CW), statistics (average, standard deviation, count), and paw identity (HL, HR, FL, and FR). [0330] In some embodiments, the at least one processor 360 can use the Gait Features for Phenotypic Analysis. In having calculated the Gait features, the at least one processor 360 can use a DRFA tool to define the phenotypic separation of a modification and a corresponding suitable control (e.g., a model of pain in rats, a model of a neurodegenerative disorder in mice, or a model of autism spectrum disorder).--, in [0329]-[0330]; also see LIN: e.g., -- The length information can be obtained from the user. For example, it may send a prompt for asking the user to generate a long captioning information or a short captioning information, the user can give corresponding instructions according to the prompt. The length information can also be obtained by analyzing the video. When the video is a real-time captured video, it may determine the current application scenario by analyzing the video, and different length information can be determined for different application scenarios.--, in [0155], and, -- When the captioning information is generated by performing encoding, it may also generate a text caption of the video that better meets the requirements of the user by acquiring information about the captioning information that the user expects to generate. For example, during the user driving, it may collect real-time video in front of the user's line of sight and analyze the video, so as to give the user a corresponding reminder by analyzing the generated captioning information of the video or play the captioning information to the user when the user needs to be prompted, such as when there is a potential danger ahead.--, in [0366]);
predict at least one action performed in the generated output representation using an action prediction model, wherein the at least one action comprises at least one of an activity, a function, and a movement corresponding to the real-time event (see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]; and, -- [0097] Compared with the spatial scene graph, the spatial-temporal scene graph adds the relationship between objects (i.e., targets) in the temporal dimension, which can better describe the spatial-temporal information of the video. In addition, the spatial-temporal scene graph can further contain the action information of the target corresponding to the temporal edge (which will be described hereinafter) to improve the accuracy of the action captioning.--, in [0097], and [0113]-[0116]; and, -- FIG. 31 illustrates a schematic flowchart of a method for generating video captioning information of the present disclosure. The 3D visual feature encoder (i.e., spatial-temporal feature extraction network), regional encoder, and semantic encoder (i.e., the semantic prediction network) shown in the figure respectively are encoders for extracting local visual features (local features shown in the figure), spatial-temporal visual features, and semantic features of respective target regions in each frame of video. Based on the local visual features, a spatial-temporal scene graph of each frame of the images can be constructed. Then, the graph convolution features (the updated local features shown in the figure) can be obtained through the graph convolution network. In this example, the 3D visual feature encoder can also be used to extract the spatial-temporal visual features of the video, and the semantic encoder can be used to extract the semantic features of the video. For the obtained spatial-temporal visual features, semantic features and graph convolution features, the feature selection network performs feature selection on various features, that is, determines the weight of each feature.--, in [0367]).
output the predicted at least one action on a user interface of a user device (see BRUNNER: e.g., --Some aspects of social interaction may occur when two subjects are in close proximity, creating occlusion from view from any unique camera setting. Quick movements of the nose may indicate an exploration of the social counterpart. Behavioral platforms consistent with disclosed embodiments can support camera mounting, resolution, and sampling rates that can enable automatic extraction of features corresponding to minute aspects of social behavior. In contrast, data acquired by conventional platforms may lack sufficient detail for automatic extraction and therefore may require manual extraction of such features. In addition, the approach to modelling the temporal aspects of all measured features may capture the complex temporal dynamics of such a social encounter. [0244] Consistent with disclosed embodiments, an analysis considering the temporal aspect of individual behavior and interactions between subjects may be more accurate and/or may provide more information than an analysis that summarizes such behavior or interactions over a period of time, such as the duration of a social encounter. For example, a temporal analysis using autocorrelation and lag analysis may indicate how the autocorrelation structure of natural behavior, or higher-order features of a subject change (e.g., changes in motifs, states, domains, or the like) as a result of the simple presence of a social counterpart.--, in [0243]; and, -- The segmentation and identification information for each video frame can be fundamental data to calculate the gait features. Given that the video image data is a 2D array, it can be summarized and converted into a tabular data format for further machine learning or statistical analysis steps. Tabulated data for each frame with the detected body part can be used as the source of information. For example, the center of the area can give the coordinate of the body or detected paws. The area and perimeter of the body area can provide shape information.--, in [0323]-[0324]; and, -- with regard to gait features summary, the at least one processor 360 may provide a summary by summarizing the gait features over multiple cycles. Features can be averaged separately for the different behaviors (all, other, walk), directions (CCW, CW), statistics (average, standard deviation, count), and paw identity (HL, HR, FL, and FR). [0330] In some embodiments, the at least one processor 360 can use the Gait Features for Phenotypic Analysis. In having calculated the Gait features, the at least one processor 360 can use a DRFA tool to define the phenotypic separation of a modification and a corresponding suitable control (e.g., a model of pain in rats, a model of a neurodegenerative disorder in mice, or a model of autism spectrum disorder).--, in [0329]-[0330]; also see LIN: e.g., -- The length information can be obtained from the user. For example, it may send a prompt for asking the user to generate a long captioning information or a short captioning information, the user can give corresponding instructions according to the prompt. The length information can also be obtained by analyzing the video. When the video is a real-time captured video, it may determine the current application scenario by analyzing the video, and different length information can be determined for different application scenarios.--, in [0155], and, -- When the captioning information is generated by performing encoding, it may also generate a text caption of the video that better meets the requirements of the user by acquiring information about the captioning information that the user expects to generate. For example, during the user driving, it may collect real-time video in front of the user's line of sight and analyze the video, so as to give the user a corresponding reminder by analyzing the generated captioning information of the video or play the captioning information to the user when the user needs to be prompted, such as when there is a potential danger ahead.--, in [0366]).
Re Claim 2, HASKIN as modified by BRUNNER and LIN further disclose validate a model performance of the action prediction model based on key performance factors, wherein the key performance factors comprise a data sensitivity factor, a data specificity factor, and a ground truth level of the action prediction model (see BRUNNER: e.g., --the thermal video processing may include the steps of image calibration, eye segmentation, feature extraction, and validation.--, in [0402]; -- [0469] Next, a classifier was trained for both class and subclass analyses using labels of drugs that are, for example, without limitation, currently prescribed or have been clinically validated for a specific therapeutic indication.--, in [0469], and, --drugs belonging in the same class of the correct drug may often be among the top-ranking predictions. For instance, the top two predictions for citalopram, in the previously described validation analysis--, in [0677]); and
tune the action prediction model to generate an updated action based on results of validation (see BRUNNER: e.g., --the thermal video processing may include the steps of image calibration, eye segmentation, feature extraction, and validation.--, in [0402]; -- [0469] Next, a classifier was trained for both class and subclass analyses using labels of drugs that are, for example, without limitation, currently prescribed or have been clinically validated for a specific therapeutic indication.--, in [0469], and, --drugs belonging in the same class of the correct drug may often be among the top-ranking predictions. For instance, the top two predictions for citalopram, in the previously described validation analysis--, in [0677]).
Re Claim 3, HASKIN as modified by BRUNNER and LIN further disclose wherein to process the obtained set of multimedia data feeds using a plurality of model hyperparameters, the processor is configured to:
identify a type of multimedia data obtained by analyzing a file format, a data size, and 10 contents of multimedia data (see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, -- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; also see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]); and,
select at least one appropriate processing model for processing the obtained set of multimedia data feeds based on the identified type of multimedia data, and the identified type of objects, the position of objects, the gestures performed within the obtained set of multimedia data feeds, the text data, and the audio data (see HASKIN: e.g., -- Streaming media is multimedia that is constantly received by and presented to an end-user while being delivered by a provider. … Live streaming refers to content delivered live over the Internet, and requires a form of source media (e.g. a video camera, an audio interface, screen capture software), an encoder to digitize the content, a media publisher, and a content delivery network to distribute and deliver the content. Streaming content may be according to, compatible with, or based on, IETF RFC 2550 entitled: “RTP: A Transport Protocol for Real-Time Applications”, IETF RFC 4587 entitled: “RTP Payload Format for H.261 Video Streams”, or IETF RFC 2326 entitled: “Real Time Streaming Protocol (RTSP)”--, in lines 10-21, page 43; also see: -- The detection mechanism is typically an electrical circuit including a re-triggerable multivibrator or another functionally similar component. Image. A digital image is a numeric representation (normally binary) of a two- dimensional image. Depending on whether the image resolution is fixed, it may be of a vector or raster type. Raster images have a finite set of digital values, called picture elements or pixels. The digital image contains a fixed number of rows and columns of pixels, which are the smallest individual element in an image, holding quantized values that represent the brightness of a given color at any specific point. Typically, the pixels are stored in computer memory as a raster image or raster map, a two-dimensional array of small integers, where these values are commonly transmitted or stored in a compressed form. The raster images can be created by a variety of input devices and techniques, such as digital cameras, scanners, coordinate-measuring machines, seismographic profiling, airborne radar, and more.--, in lines 1-11, page 11; and, -- Video content analysis transforms the audio and image stream into a set of semantically meaningful representations. The ultimate goal is to extract structural and semantic content automatically, without any human intervention, at least for limited types of video domains. Algorithms to perform content analysis include those for detecting objects in video, recognizing specific objects, persons, locations, detecting dynamic events in video, associating keywords with image regions or motion. VCA is used in a wide range of domains including entertainment, health-care, retail, automotive, transport, home automation, flame and smoke detection, safety and security. The algorithms can be implemented as software on general purpose machines, or as hardware in specialized video processing units. Many different functionalities can be implemented in VCA. Video Motion Detection is one of the simpler forms where motion is detected with regard to a fixed background scene. More advanced functionalities include video tracking and egomotion estimation. Based on the internal representation that VCA generates in the machine, it is possible to build other functionalities, such as identification, behavior analysis or other forms of situation awareness. VCA typically relies on good input video, so it is commonly combined with video enhancement technologies such as video denoising, image stabilization, unsharp masking and super- resolution--, in line 27, page 31 through line 15, page 32; also see: --YOLO is implemented as a CNN…. The YOLO approach to object detection describing frame object detection as a regression problem to spatially separated bounding boxes and associated class probabilities is described in an article authored by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi, published 9 May 2016 and entitled: “You Only Look Once: Unified, Real-Time Object Detection”, which is incorporated in its entirety for all purposes as if fully set forth herein. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance. The base YOLO model processes images in real- time at 45 frames per second while a smaller version of the network, Fast YOLO, processes an astounding 155 frames per second while still achieving double the mAP of other real-time detectors.--, in lines 10-32, page 60; --As for object detection accuracy, it achieves a mean Average Precision (mAP) of 64.16% for PASCAL VOC 2007 dataset that is only 2.63% lower than the mAP of the same YOLO network with full precision. R-CNN. Regions with CNN features (R-CNN) family is a family of machine learning models used to bypass the problem of selecting a huge number of regions. The R-CNN uses selective search to extract just 2000 regions from the image, referred to as region proposals. Then, instead of trying to classify a huge number of regions, only 2000 regions are handled. These 2000 region proposals are generated using a selective search algorithm, that includes Generating initial sub-segmentation for generating many candidate regions, using greedy algorithm to recursively combine similar regions into larger ones, and using the generated regions to produce the final candidate region proposals. These 2000 candidate region proposals are warped into a square and fed into a convolutional neural network that produces a 4096- dimensional feature vector as output.—, in lines 15-28, page 62; and,
--As in the original R-CNN, the Fast R-CNN uses Selective Search to generate its region proposals. While Fast R-CNN used Selective Search to generate ROIs, Faster R-CNN integrates the ROI generation into the neural network itself. Mask R-CNN adds instance segmentation, and also replaced ROIPooling with a new method called ROIAlign, which can represent fractions of a pixel, and Mesh R-CNN adds the ability to generate a 3D mesh from a 2D image. R-CNN and Fast R-CNN are primarily image classifier networks which are used for object detection by using Region Proposal method to generate potential bounding boxes in an image, run the classifier on these boxes, and after classification, perform post processing to tighten the boundaries of the bounding boxes and remove duplicates. Regions with CNN features (R-CNN) that combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost--, in lines 15-28, page 63;
-- A Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals, is described in an article authored by Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, published 2015, entitled: “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal networks”, which is incorporated in its entirety for all purposes as if fully set forth herein. State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. An RPN is a fully- convolutional network that simultaneously predicts object bounds and objectness scores at each position. RPNs are trained end-to-end to generate high quality region proposals, which are used by Fast R-CNN for detection. With a simple alternating optimization, RPN and Fast R-CNN can be trained to share convolutional features. For the very deep VGG-16 model, a described detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state- of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image. Code is available at https://github.com/ShaoqingRen/faster_rcnn. RetinaNet. RetinaNet is one of the one-stage object detection models that has proven to work well with dense and small-scale objects, that has become a popular object detection model to be used with aerial and satellite imagery. RetinaNet has been formed by making two improvements over existing single stage object detection models - Feature Pyramid Networks (FPN) and Focal Loss. Traditionally, in computer vision, featurized image pyramids have been used to detect objects with varying scales in an image. Featurized image pyramids are feature pyramids built upon image pyramids, where an image is subsampled into lower resolution and smaller size images (thus, forming a pyramid). Hand-engineered features are then extracted from each layer in the pyramid to detect the objects, which makes the pyramid scale-invariant.--, in line 31, page 64 through line 35, page 65; and, -- applying these mobile models to object detection in a novel framework referred to as SSDLite, and further demonstrates how to build mobile semantic segmentation models through a reduced form of DeepLabv3 (referred to as Mobile DeepLabv3), is based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers. The intermediate expansion layer uses lightweight depth-wise convolutions to filter features as a source of non-linearity. The scheme allows for decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis.--, in line 10, page 70 through line 8, page 71; also see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, -- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; also see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]); and
process the obtained set of multimedia data feeds using the selected at least one appropriate processing model (see HASKIN: e.g., -- Streaming media is multimedia that is constantly received by and presented to an end-user while being delivered by a provider. … Live streaming refers to content delivered live over the Internet, and requires a form of source media (e.g. a video camera, an audio interface, screen capture software), an encoder to digitize the content, a media publisher, and a content delivery network to distribute and deliver the content. Streaming content may be according to, compatible with, or based on, IETF RFC 2550 entitled: “RTP: A Transport Protocol for Real-Time Applications”, IETF RFC 4587 entitled: “RTP Payload Format for H.261 Video Streams”, or IETF RFC 2326 entitled: “Real Time Streaming Protocol (RTSP)”--, in lines 10-21, page 43; also see: -- The detection mechanism is typically an electrical circuit including a re-triggerable multivibrator or another functionally similar component. Image. A digital image is a numeric representation (normally binary) of a two- dimensional image. Depending on whether the image resolution is fixed, it may be of a vector or raster type. Raster images have a finite set of digital values, called picture elements or pixels. The digital image contains a fixed number of rows and columns of pixels, which are the smallest individual element in an image, holding quantized values that represent the brightness of a given color at any specific point. Typically, the pixels are stored in computer memory as a raster image or raster map, a two-dimensional array of small integers, where these values are commonly transmitted or stored in a compressed form. The raster images can be created by a variety of input devices and techniques, such as digital cameras, scanners, coordinate-measuring machines, seismographic profiling, airborne radar, and more.--, in lines 1-11, page 11; and, -- Video content analysis transforms the audio and image stream into a set of semantically meaningful representations. The ultimate goal is to extract structural and semantic content automatically, without any human intervention, at least for limited types of video domains. Algorithms to perform content analysis include those for detecting objects in video, recognizing specific objects, persons, locations, detecting dynamic events in video, associating keywords with image regions or motion. VCA is used in a wide range of domains including entertainment, health-care, retail, automotive, transport, home automation, flame and smoke detection, safety and security. The algorithms can be implemented as software on general purpose machines, or as hardware in specialized video processing units. Many different functionalities can be implemented in VCA. Video Motion Detection is one of the simpler forms where motion is detected with regard to a fixed background scene. More advanced functionalities include video tracking and egomotion estimation. Based on the internal representation that VCA generates in the machine, it is possible to build other functionalities, such as identification, behavior analysis or other forms of situation awareness. VCA typically relies on good input video, so it is commonly combined with video enhancement technologies such as video denoising, image stabilization, unsharp masking and super- resolution--, in line 27, page 31 through line 15, page 32; also see: --YOLO is implemented as a CNN…. The YOLO approach to object detection describing frame object detection as a regression problem to spatially separated bounding boxes and associated class probabilities is described in an article authored by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi, published 9 May 2016 and entitled: “You Only Look Once: Unified, Real-Time Object Detection”, which is incorporated in its entirety for all purposes as if fully set forth herein. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance. The base YOLO model processes images in real- time at 45 frames per second while a smaller version of the network, Fast YOLO, processes an astounding 155 frames per second while still achieving double the mAP of other real-time detectors.--, in lines 10-32, page 60; --As for object detection accuracy, it achieves a mean Average Precision (mAP) of 64.16% for PASCAL VOC 2007 dataset that is only 2.63% lower than the mAP of the same YOLO network with full precision. R-CNN. Regions with CNN features (R-CNN) family is a family of machine learning models used to bypass the problem of selecting a huge number of regions. The R-CNN uses selective search to extract just 2000 regions from the image, referred to as region proposals. Then, instead of trying to classify a huge number of regions, only 2000 regions are handled. These 2000 region proposals are generated using a selective search algorithm, that includes Generating initial sub-segmentation for generating many candidate regions, using greedy algorithm to recursively combine similar regions into larger ones, and using the generated regions to produce the final candidate region proposals. These 2000 candidate region proposals are warped into a square and fed into a convolutional neural network that produces a 4096- dimensional feature vector as output.—, in lines 15-28, page 62; and,
--As in the original R-CNN, the Fast R-CNN uses Selective Search to generate its region proposals. While Fast R-CNN used Selective Search to generate ROIs, Faster R-CNN integrates the ROI generation into the neural network itself. Mask R-CNN adds instance segmentation, and also replaced ROIPooling with a new method called ROIAlign, which can represent fractions of a pixel, and Mesh R-CNN adds the ability to generate a 3D mesh from a 2D image. R-CNN and Fast R-CNN are primarily image classifier networks which are used for object detection by using Region Proposal method to generate potential bounding boxes in an image, run the classifier on these boxes, and after classification, perform post processing to tighten the boundaries of the bounding boxes and remove duplicates. Regions with CNN features (R-CNN) that combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost--, in lines 15-28, page 63;
-- A Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals, is described in an article authored by Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, published 2015, entitled: “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal networks”, which is incorporated in its entirety for all purposes as if fully set forth herein. State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. An RPN is a fully- convolutional network that simultaneously predicts object bounds and objectness scores at each position. RPNs are trained end-to-end to generate high quality region proposals, which are used by Fast R-CNN for detection. With a simple alternating optimization, RPN and Fast R-CNN can be trained to share convolutional features. For the very deep VGG-16 model, a described detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state- of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image. Code is available at https://github.com/ShaoqingRen/faster_rcnn. RetinaNet. RetinaNet is one of the one-stage object detection models that has proven to work well with dense and small-scale objects, that has become a popular object detection model to be used with aerial and satellite imagery. RetinaNet has been formed by making two improvements over existing single stage object detection models - Feature Pyramid Networks (FPN) and Focal Loss. Traditionally, in computer vision, featurized image pyramids have been used to detect objects with varying scales in an image. Featurized image pyramids are feature pyramids built upon image pyramids, where an image is subsampled into lower resolution and smaller size images (thus, forming a pyramid). Hand-engineered features are then extracted from each layer in the pyramid to detect the objects, which makes the pyramid scale-invariant.--, in line 31, page 64 through line 35, page 65; and, -- applying these mobile models to object detection in a novel framework referred to as SSDLite, and further demonstrates how to build mobile semantic segmentation models through a reduced form of DeepLabv3 (referred to as Mobile DeepLabv3), is based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers. The intermediate expansion layer uses lightweight depth-wise convolutions to filter features as a source of non-linearity. The scheme allows for decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis.--, in line 10, page 70 through line 8, page 71; also see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, -- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; also see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]).
Re Claim 4, HASKIN as modified by BRUNNER and LIN further disclose une the plurality of model hyperparameters based on the selected at least one appropriate processing model, wherein the selected at least one appropriate processing model comprises a computer vision model and an audio model, wherein the computer vision model comprises at least one of an object detection, an object tracking, a person detection, a person tracking, a semantic segmentation, a semantic compression, a multi-camera person recognition, and a multi-camera object recognition, wherein the audio model comprises a noise reduction, a speech detection, and a speech diarization (see HASKIN: e.g., -- Streaming media is multimedia that is constantly received by and presented to an end-user while being delivered by a provider. … Live streaming refers to content delivered live over the Internet, and requires a form of source media (e.g. a video camera, an audio interface, screen capture software), an encoder to digitize the content, a media publisher, and a content delivery network to distribute and deliver the content. Streaming content may be according to, compatible with, or based on, IETF RFC 2550 entitled: “RTP: A Transport Protocol for Real-Time Applications”, IETF RFC 4587 entitled: “RTP Payload Format for H.261 Video Streams”, or IETF RFC 2326 entitled: “Real Time Streaming Protocol (RTSP)”--, in lines 10-21, page 43; also see: -- The detection mechanism is typically an electrical circuit including a re-triggerable multivibrator or another functionally similar component. Image. A digital image is a numeric representation (normally binary) of a two- dimensional image. Depending on whether the image resolution is fixed, it may be of a vector or raster type. Raster images have a finite set of digital values, called picture elements or pixels. The digital image contains a fixed number of rows and columns of pixels, which are the smallest individual element in an image, holding quantized values that represent the brightness of a given color at any specific point. Typically, the pixels are stored in computer memory as a raster image or raster map, a two-dimensional array of small integers, where these values are commonly transmitted or stored in a compressed form. The raster images can be created by a variety of input devices and techniques, such as digital cameras, scanners, coordinate-measuring machines, seismographic profiling, airborne radar, and more.--, in lines 1-11, page 11; and, -- Video content analysis transforms the audio and image stream into a set of semantically meaningful representations. The ultimate goal is to extract structural and semantic content automatically, without any human intervention, at least for limited types of video domains. Algorithms to perform content analysis include those for detecting objects in video, recognizing specific objects, persons, locations, detecting dynamic events in video, associating keywords with image regions or motion. VCA is used in a wide range of domains including entertainment, health-care, retail, automotive, transport, home automation, flame and smoke detection, safety and security. The algorithms can be implemented as software on general purpose machines, or as hardware in specialized video processing units. Many different functionalities can be implemented in VCA. Video Motion Detection is one of the simpler forms where motion is detected with regard to a fixed background scene. More advanced functionalities include video tracking and egomotion estimation. Based on the internal representation that VCA generates in the machine, it is possible to build other functionalities, such as identification, behavior analysis or other forms of situation awareness. VCA typically relies on good input video, so it is commonly combined with video enhancement technologies such as video denoising, image stabilization, unsharp masking and super- resolution--, in line 27, page 31 through line 15, page 32; also see: --YOLO is implemented as a CNN…. The YOLO approach to object detection describing frame object detection as a regression problem to spatially separated bounding boxes and associated class probabilities is described in an article authored by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi, published 9 May 2016 and entitled: “You Only Look Once: Unified, Real-Time Object Detection”, which is incorporated in its entirety for all purposes as if fully set forth herein. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance. The base YOLO model processes images in real- time at 45 frames per second while a smaller version of the network, Fast YOLO, processes an astounding 155 frames per second while still achieving double the mAP of other real-time detectors.--, in lines 10-32, page 60; --As for object detection accuracy, it achieves a mean Average Precision (mAP) of 64.16% for PASCAL VOC 2007 dataset that is only 2.63% lower than the mAP of the same YOLO network with full precision. R-CNN. Regions with CNN features (R-CNN) family is a family of machine learning models used to bypass the problem of selecting a huge number of regions. The R-CNN uses selective search to extract just 2000 regions from the image, referred to as region proposals. Then, instead of trying to classify a huge number of regions, only 2000 regions are handled. These 2000 region proposals are generated using a selective search algorithm, that includes Generating initial sub-segmentation for generating many candidate regions, using greedy algorithm to recursively combine similar regions into larger ones, and using the generated regions to produce the final candidate region proposals. These 2000 candidate region proposals are warped into a square and fed into a convolutional neural network that produces a 4096- dimensional feature vector as output.—, in lines 15-28, page 62; and,
--As in the original R-CNN, the Fast R-CNN uses Selective Search to generate its region proposals. While Fast R-CNN used Selective Search to generate ROIs, Faster R-CNN integrates the ROI generation into the neural network itself. Mask R-CNN adds instance segmentation, and also replaced ROIPooling with a new method called ROIAlign, which can represent fractions of a pixel, and Mesh R-CNN adds the ability to generate a 3D mesh from a 2D image. R-CNN and Fast R-CNN are primarily image classifier networks which are used for object detection by using Region Proposal method to generate potential bounding boxes in an image, run the classifier on these boxes, and after classification, perform post processing to tighten the boundaries of the bounding boxes and remove duplicates. Regions with CNN features (R-CNN) that combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost--, in lines 15-28, page 63;
-- A Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals, is described in an article authored by Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, published 2015, entitled: “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal networks”, which is incorporated in its entirety for all purposes as if fully set forth herein. State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. An RPN is a fully- convolutional network that simultaneously predicts object bounds and objectness scores at each position. RPNs are trained end-to-end to generate high quality region proposals, which are used by Fast R-CNN for detection. With a simple alternating optimization, RPN and Fast R-CNN can be trained to share convolutional features. For the very deep VGG-16 model, a described detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state- of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image. Code is available at https://github.com/ShaoqingRen/faster_rcnn. RetinaNet. RetinaNet is one of the one-stage object detection models that has proven to work well with dense and small-scale objects, that has become a popular object detection model to be used with aerial and satellite imagery. RetinaNet has been formed by making two improvements over existing single stage object detection models - Feature Pyramid Networks (FPN) and Focal Loss. Traditionally, in computer vision, featurized image pyramids have been used to detect objects with varying scales in an image. Featurized image pyramids are feature pyramids built upon image pyramids, where an image is subsampled into lower resolution and smaller size images (thus, forming a pyramid). Hand-engineered features are then extracted from each layer in the pyramid to detect the objects, which makes the pyramid scale-invariant.--, in line 31, page 64 through line 35, page 65; and, -- applying these mobile models to object detection in a novel framework referred to as SSDLite, and further demonstrates how to build mobile semantic segmentation models through a reduced form of DeepLabv3 (referred to as Mobile DeepLabv3), is based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers. The intermediate expansion layer uses lightweight depth-wise convolutions to filter features as a source of non-linearity. The scheme allows for decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis.--, in line 10, page 70 through line 8, page 71; also see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, -- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; also see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]);
retrain a vision encoder model based on the tuned plurality of model hyperparameters (see HASKIN: e.g., -- Streaming media is multimedia that is constantly received by and presented to an end-user while being delivered by a provider. … Live streaming refers to content delivered live over the Internet, and requires a form of source media (e.g. a video camera, an audio interface, screen capture software), an encoder to digitize the content, a media publisher, and a content delivery network to distribute and deliver the content. Streaming content may be according to, compatible with, or based on, IETF RFC 2550 entitled: “RTP: A Transport Protocol for Real-Time Applications”, IETF RFC 4587 entitled: “RTP Payload Format for H.261 Video Streams”, or IETF RFC 2326 entitled: “Real Time Streaming Protocol (RTSP)”--, in lines 10-21, page 43; also see: -- The detection mechanism is typically an electrical circuit including a re-triggerable multivibrator or another functionally similar component. Image. A digital image is a numeric representation (normally binary) of a two- dimensional image. Depending on whether the image resolution is fixed, it may be of a vector or raster type. Raster images have a finite set of digital values, called picture elements or pixels. The digital image contains a fixed number of rows and columns of pixels, which are the smallest individual element in an image, holding quantized values that represent the brightness of a given color at any specific point. Typically, the pixels are stored in computer memory as a raster image or raster map, a two-dimensional array of small integers, where these values are commonly transmitted or stored in a compressed form. The raster images can be created by a variety of input devices and techniques, such as digital cameras, scanners, coordinate-measuring machines, seismographic profiling, airborne radar, and more.--, in lines 1-11, page 11; and, -- Video content analysis transforms the audio and image stream into a set of semantically meaningful representations. The ultimate goal is to extract structural and semantic content automatically, without any human intervention, at least for limited types of video domains. Algorithms to perform content analysis include those for detecting objects in video, recognizing specific objects, persons, locations, detecting dynamic events in video, associating keywords with image regions or motion. VCA is used in a wide range of domains including entertainment, health-care, retail, automotive, transport, home automation, flame and smoke detection, safety and security. The algorithms can be implemented as software on general purpose machines, or as hardware in specialized video processing units. Many different functionalities can be implemented in VCA. Video Motion Detection is one of the simpler forms where motion is detected with regard to a fixed background scene. More advanced functionalities include video tracking and egomotion estimation. Based on the internal representation that VCA generates in the machine, it is possible to build other functionalities, such as identification, behavior analysis or other forms of situation awareness. VCA typically relies on good input video, so it is commonly combined with video enhancement technologies such as video denoising, image stabilization, unsharp masking and super- resolution--, in line 27, page 31 through line 15, page 32; also see: --YOLO is implemented as a CNN…. The YOLO approach to object detection describing frame object detection as a regression problem to spatially separated bounding boxes and associated class probabilities is described in an article authored by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi, published 9 May 2016 and entitled: “You Only Look Once: Unified, Real-Time Object Detection”, which is incorporated in its entirety for all purposes as if fully set forth herein. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance. The base YOLO model processes images in real- time at 45 frames per second while a smaller version of the network, Fast YOLO, processes an astounding 155 frames per second while still achieving double the mAP of other real-time detectors.--, in lines 10-32, page 60; --As for object detection accuracy, it achieves a mean Average Precision (mAP) of 64.16% for PASCAL VOC 2007 dataset that is only 2.63% lower than the mAP of the same YOLO network with full precision. R-CNN. Regions with CNN features (R-CNN) family is a family of machine learning models used to bypass the problem of selecting a huge number of regions. The R-CNN uses selective search to extract just 2000 regions from the image, referred to as region proposals. Then, instead of trying to classify a huge number of regions, only 2000 regions are handled. These 2000 region proposals are generated using a selective search algorithm, that includes Generating initial sub-segmentation for generating many candidate regions, using greedy algorithm to recursively combine similar regions into larger ones, and using the generated regions to produce the final candidate region proposals. These 2000 candidate region proposals are warped into a square and fed into a convolutional neural network that produces a 4096- dimensional feature vector as output.—, in lines 15-28, page 62; and,
--As in the original R-CNN, the Fast R-CNN uses Selective Search to generate its region proposals. While Fast R-CNN used Selective Search to generate ROIs, Faster R-CNN integrates the ROI generation into the neural network itself. Mask R-CNN adds instance segmentation, and also replaced ROIPooling with a new method called ROIAlign, which can represent fractions of a pixel, and Mesh R-CNN adds the ability to generate a 3D mesh from a 2D image. R-CNN and Fast R-CNN are primarily image classifier networks which are used for object detection by using Region Proposal method to generate potential bounding boxes in an image, run the classifier on these boxes, and after classification, perform post processing to tighten the boundaries of the bounding boxes and remove duplicates. Regions with CNN features (R-CNN) that combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost--, in lines 15-28, page 63;
-- A Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals, is described in an article authored by Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, published 2015, entitled: “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal networks”, which is incorporated in its entirety for all purposes as if fully set forth herein. State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. An RPN is a fully- convolutional network that simultaneously predicts object bounds and objectness scores at each position. RPNs are trained end-to-end to generate high quality region proposals, which are used by Fast R-CNN for detection. With a simple alternating optimization, RPN and Fast R-CNN can be trained to share convolutional features. For the very deep VGG-16 model, a described detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state- of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image. Code is available at https://github.com/ShaoqingRen/faster_rcnn. RetinaNet. RetinaNet is one of the one-stage object detection models that has proven to work well with dense and small-scale objects, that has become a popular object detection model to be used with aerial and satellite imagery. RetinaNet has been formed by making two improvements over existing single stage object detection models - Feature Pyramid Networks (FPN) and Focal Loss. Traditionally, in computer vision, featurized image pyramids have been used to detect objects with varying scales in an image. Featurized image pyramids are feature pyramids built upon image pyramids, where an image is subsampled into lower resolution and smaller size images (thus, forming a pyramid). Hand-engineered features are then extracted from each layer in the pyramid to detect the objects, which makes the pyramid scale-invariant.--, in line 31, page 64 through line 35, page 65; and, -- applying these mobile models to object detection in a novel framework referred to as SSDLite, and further demonstrates how to build mobile semantic segmentation models through a reduced form of DeepLabv3 (referred to as Mobile DeepLabv3), is based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers. The intermediate expansion layer uses lightweight depth-wise convolutions to filter features as a source of non-linearity. The scheme allows for decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis.--, in line 10, page 70 through line 8, page 71; also see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, -- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; also see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]).
Re Claim 5, HASKIN as modified by BRUNNER and LIN further disclose wherein to obtain the at least one input prompt corresponding to the set of multimedia data feeds from at least one input source the processor is configured to: obtain text prompts from at least one input source at real-time based on a type of the set of multimedia data feeds, wherein the at least one input source comprises one of a user input, and
a model input (see HASKIN: e.g., -- Similarly, the receiving path comprises a received channel in the RF spectrum, down-converted to the baseband (IF) wherein the data is then extracted. In order to support multiple devices and use a permanent solution, a Wireless Access Point (WAP) is typically used… Authentication in general, and a WAP authentication in particular, is used as the basis for authorization, which determines whether a privilege may be granted to a particular user or process, privacy, which keeps information from becoming known to non-participants, and non-repudiation, which is the inability to deny having done something that was authorized to be done based on the authentication. An authentication in general, and a WAP authentication in particular, may use an authentication server that provides a network service that applications may use to authenticate the credentials, usually account names and passwords of their users. When a client submits a valid set of credentials, it receives a cryptographic ticket that it can subsequently be used to access various services. Authentication algorithms include passwords, Kerberos, and public key encryption. Prior art technologies for data networking may be based on single carrier modulation techniques, such as AM (Amplitude Modulation), FM (Frequency Modulation), and PM (Phase Modulation), as well as bit encoding techniques such as QAM (Quadrature Amplitude Modulation) and QPSK (Quadrature Phase Shift Keying).--, in line 32, page 23 through line 14, page 24; also see LIN: e.g., -- The length information can be obtained from the user. For example, it may send a prompt for asking the user to generate a long captioning information or a short captioning information, the user can give corresponding instructions according to the prompt. The length information can also be obtained by analyzing the video. When the video is a real-time captured video, it may determine the current application scenario by analyzing the video, and different length information can be determined for different application scenarios.--, in [0155], and, -- When the captioning information is generated by performing encoding, it may also generate a text caption of the video that better meets the requirements of the user by acquiring information about the captioning information that the user expects to generate. For example, during the user driving, it may collect real-time video in front of the user's line of sight and analyze the video, so as to give the user a corresponding reminder by analyzing the generated captioning information of the video or play the captioning information to the user when the user needs to be prompted, such as when there is a potential danger ahead.--, in [0366]);
Re Claim 6, HASKIN as modified by BRUNNER and LIN further disclose wherein to process the obtained set of multimedia data feeds using the plurality of model hyperparameters, the processor is configured to:
identify an event of interest within the set of multimedia data feeds captured from the plurality of image capturing devices (see HASKIN: e.g., -- Streaming media is multimedia that is constantly received by and presented to an end-user while being delivered by a provider. … Live streaming refers to content delivered live over the Internet, and requires a form of source media (e.g. a video camera, an audio interface, screen capture software), an encoder to digitize the content, a media publisher, and a content delivery network to distribute and deliver the content. Streaming content may be according to, compatible with, or based on, IETF RFC 2550 entitled: “RTP: A Transport Protocol for Real-Time Applications”, IETF RFC 4587 entitled: “RTP Payload Format for H.261 Video Streams”, or IETF RFC 2326 entitled: “Real Time Streaming Protocol (RTSP)”--, in lines 10-21, page 43; also see: -- The detection mechanism is typically an electrical circuit including a re-triggerable multivibrator or another functionally similar component. Image. A digital image is a numeric representation (normally binary) of a two- dimensional image. Depending on whether the image resolution is fixed, it may be of a vector or raster type. Raster images have a finite set of digital values, called picture elements or pixels. The digital image contains a fixed number of rows and columns of pixels, which are the smallest individual element in an image, holding quantized values that represent the brightness of a given color at any specific point. Typically, the pixels are stored in computer memory as a raster image or raster map, a two-dimensional array of small integers, where these values are commonly transmitted or stored in a compressed form. The raster images can be created by a variety of input devices and techniques, such as digital cameras, scanners, coordinate-measuring machines, seismographic profiling, airborne radar, and more.--, in lines 1-11, page 11; and, -- Video content analysis transforms the audio and image stream into a set of semantically meaningful representations. The ultimate goal is to extract structural and semantic content automatically, without any human intervention, at least for limited types of video domains. Algorithms to perform content analysis include those for detecting objects in video, recognizing specific objects, persons, locations, detecting dynamic events in video, associating keywords with image regions or motion. VCA is used in a wide range of domains including entertainment, health-care, retail, automotive, transport, home automation, flame and smoke detection, safety and security. The algorithms can be implemented as software on general purpose machines, or as hardware in specialized video processing units. Many different functionalities can be implemented in VCA. Video Motion Detection is one of the simpler forms where motion is detected with regard to a fixed background scene. More advanced functionalities include video tracking and egomotion estimation. Based on the internal representation that VCA generates in the machine, it is possible to build other functionalities, such as identification, behavior analysis or other forms of situation awareness. VCA typically relies on good input video, so it is commonly combined with video enhancement technologies such as video denoising, image stabilization, unsharp masking and super- resolution--, in line 27, page 31 through line 15, page 32; also see: --YOLO is implemented as a CNN…. The YOLO approach to object detection describing frame object detection as a regression problem to spatially separated bounding boxes and associated class probabilities is described in an article authored by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi, published 9 May 2016 and entitled: “You Only Look Once: Unified, Real-Time Object Detection”, which is incorporated in its entirety for all purposes as if fully set forth herein. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance. The base YOLO model processes images in real- time at 45 frames per second while a smaller version of the network, Fast YOLO, processes an astounding 155 frames per second while still achieving double the mAP of other real-time detectors.--, in lines 10-32, page 60; --As for object detection accuracy, it achieves a mean Average Precision (mAP) of 64.16% for PASCAL VOC 2007 dataset that is only 2.63% lower than the mAP of the same YOLO network with full precision. R-CNN. Regions with CNN features (R-CNN) family is a family of machine learning models used to bypass the problem of selecting a huge number of regions. The R-CNN uses selective search to extract just 2000 regions from the image, referred to as region proposals. Then, instead of trying to classify a huge number of regions, only 2000 regions are handled. These 2000 region proposals are generated using a selective search algorithm, that includes Generating initial sub-segmentation for generating many candidate regions, using greedy algorithm to recursively combine similar regions into larger ones, and using the generated regions to produce the final candidate region proposals. These 2000 candidate region proposals are warped into a square and fed into a convolutional neural network that produces a 4096- dimensional feature vector as output.—, in lines 15-28, page 62; and,
--As in the original R-CNN, the Fast R-CNN uses Selective Search to generate its region proposals. While Fast R-CNN used Selective Search to generate ROIs, Faster R-CNN integrates the ROI generation into the neural network itself. Mask R-CNN adds instance segmentation, and also replaced ROIPooling with a new method called ROIAlign, which can represent fractions of a pixel, and Mesh R-CNN adds the ability to generate a 3D mesh from a 2D image. R-CNN and Fast R-CNN are primarily image classifier networks which are used for object detection by using Region Proposal method to generate potential bounding boxes in an image, run the classifier on these boxes, and after classification, perform post processing to tighten the boundaries of the bounding boxes and remove duplicates. Regions with CNN features (R-CNN) that combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost--, in lines 15-28, page 63;
-- A Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals, is described in an article authored by Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, published 2015, entitled: “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal networks”, which is incorporated in its entirety for all purposes as if fully set forth herein. State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. An RPN is a fully- convolutional network that simultaneously predicts object bounds and objectness scores at each position. RPNs are trained end-to-end to generate high quality region proposals, which are used by Fast R-CNN for detection. With a simple alternating optimization, RPN and Fast R-CNN can be trained to share convolutional features. For the very deep VGG-16 model, a described detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state- of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image. Code is available at https://github.com/ShaoqingRen/faster_rcnn. RetinaNet. RetinaNet is one of the one-stage object detection models that has proven to work well with dense and small-scale objects, that has become a popular object detection model to be used with aerial and satellite imagery. RetinaNet has been formed by making two improvements over existing single stage object detection models - Feature Pyramid Networks (FPN) and Focal Loss. Traditionally, in computer vision, featurized image pyramids have been used to detect objects with varying scales in an image. Featurized image pyramids are feature pyramids built upon image pyramids, where an image is subsampled into lower resolution and smaller size images (thus, forming a pyramid). Hand-engineered features are then extracted from each layer in the pyramid to detect the objects, which makes the pyramid scale-invariant.--, in line 31, page 64 through line 35, page 65; and, -- applying these mobile models to object detection in a novel framework referred to as SSDLite, and further demonstrates how to build mobile semantic segmentation models through a reduced form of DeepLabv3 (referred to as Mobile DeepLabv3), is based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers. The intermediate expansion layer uses lightweight depth-wise convolutions to filter features as a source of non-linearity. The scheme allows for decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis.--, in line 10, page 70 through line 8, page 71; also see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, -- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; also see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]);
determine a plurality of patterns corresponding to the identified event of interest with respect to a plurality of time instances by correlating each media frame with a subsequent media frame of the set of multimedia data feeds (see HASKIN: e.g., -- Streaming media is multimedia that is constantly received by and presented to an end-user while being delivered by a provider. … Live streaming refers to content delivered live over the Internet, and requires a form of source media (e.g. a video camera, an audio interface, screen capture software), an encoder to digitize the content, a media publisher, and a content delivery network to distribute and deliver the content. Streaming content may be according to, compatible with, or based on, IETF RFC 2550 entitled: “RTP: A Transport Protocol for Real-Time Applications”, IETF RFC 4587 entitled: “RTP Payload Format for H.261 Video Streams”, or IETF RFC 2326 entitled: “Real Time Streaming Protocol (RTSP)”--, in lines 10-21, page 43; also see: -- The detection mechanism is typically an electrical circuit including a re-triggerable multivibrator or another functionally similar component. Image. A digital image is a numeric representation (normally binary) of a two- dimensional image. Depending on whether the image resolution is fixed, it may be of a vector or raster type. Raster images have a finite set of digital values, called picture elements or pixels. The digital image contains a fixed number of rows and columns of pixels, which are the smallest individual element in an image, holding quantized values that represent the brightness of a given color at any specific point. Typically, the pixels are stored in computer memory as a raster image or raster map, a two-dimensional array of small integers, where these values are commonly transmitted or stored in a compressed form. The raster images can be created by a variety of input devices and techniques, such as digital cameras, scanners, coordinate-measuring machines, seismographic profiling, airborne radar, and more.--, in lines 1-11, page 11; and, -- Video content analysis transforms the audio and image stream into a set of semantically meaningful representations. The ultimate goal is to extract structural and semantic content automatically, without any human intervention, at least for limited types of video domains. Algorithms to perform content analysis include those for detecting objects in video, recognizing specific objects, persons, locations, detecting dynamic events in video, associating keywords with image regions or motion. VCA is used in a wide range of domains including entertainment, health-care, retail, automotive, transport, home automation, flame and smoke detection, safety and security. The algorithms can be implemented as software on general purpose machines, or as hardware in specialized video processing units. Many different functionalities can be implemented in VCA. Video Motion Detection is one of the simpler forms where motion is detected with regard to a fixed background scene. More advanced functionalities include video tracking and egomotion estimation. Based on the internal representation that VCA generates in the machine, it is possible to build other functionalities, such as identification, behavior analysis or other forms of situation awareness. VCA typically relies on good input video, so it is commonly combined with video enhancement technologies such as video denoising, image stabilization, unsharp masking and super- resolution--, in line 27, page 31 through line 15, page 32; also see: --YOLO is implemented as a CNN…. The YOLO approach to object detection describing frame object detection as a regression problem to spatially separated bounding boxes and associated class probabilities is described in an article authored by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi, published 9 May 2016 and entitled: “You Only Look Once: Unified, Real-Time Object Detection”, which is incorporated in its entirety for all purposes as if fully set forth herein. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance. The base YOLO model processes images in real- time at 45 frames per second while a smaller version of the network, Fast YOLO, processes an astounding 155 frames per second while still achieving double the mAP of other real-time detectors.--, in lines 10-32, page 60; --As for object detection accuracy, it achieves a mean Average Precision (mAP) of 64.16% for PASCAL VOC 2007 dataset that is only 2.63% lower than the mAP of the same YOLO network with full precision. R-CNN. Regions with CNN features (R-CNN) family is a family of machine learning models used to bypass the problem of selecting a huge number of regions. The R-CNN uses selective search to extract just 2000 regions from the image, referred to as region proposals. Then, instead of trying to classify a huge number of regions, only 2000 regions are handled. These 2000 region proposals are generated using a selective search algorithm, that includes Generating initial sub-segmentation for generating many candidate regions, using greedy algorithm to recursively combine similar regions into larger ones, and using the generated regions to produce the final candidate region proposals. These 2000 candidate region proposals are warped into a square and fed into a convolutional neural network that produces a 4096- dimensional feature vector as output.—, in lines 15-28, page 62; and,
--As in the original R-CNN, the Fast R-CNN uses Selective Search to generate its region proposals. While Fast R-CNN used Selective Search to generate ROIs, Faster R-CNN integrates the ROI generation into the neural network itself. Mask R-CNN adds instance segmentation, and also replaced ROIPooling with a new method called ROIAlign, which can represent fractions of a pixel, and Mesh R-CNN adds the ability to generate a 3D mesh from a 2D image. R-CNN and Fast R-CNN are primarily image classifier networks which are used for object detection by using Region Proposal method to generate potential bounding boxes in an image, run the classifier on these boxes, and after classification, perform post processing to tighten the boundaries of the bounding boxes and remove duplicates. Regions with CNN features (R-CNN) that combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost--, in lines 15-28, page 63;
-- A Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals, is described in an article authored by Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, published 2015, entitled: “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal networks”, which is incorporated in its entirety for all purposes as if fully set forth herein. State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. An RPN is a fully- convolutional network that simultaneously predicts object bounds and objectness scores at each position. RPNs are trained end-to-end to generate high quality region proposals, which are used by Fast R-CNN for detection. With a simple alternating optimization, RPN and Fast R-CNN can be trained to share convolutional features. For the very deep VGG-16 model, a described detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state- of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image. Code is available at https://github.com/ShaoqingRen/faster_rcnn. RetinaNet. RetinaNet is one of the one-stage object detection models that has proven to work well with dense and small-scale objects, that has become a popular object detection model to be used with aerial and satellite imagery. RetinaNet has been formed by making two improvements over existing single stage object detection models - Feature Pyramid Networks (FPN) and Focal Loss. Traditionally, in computer vision, featurized image pyramids have been used to detect objects with varying scales in an image. Featurized image pyramids are feature pyramids built upon image pyramids, where an image is subsampled into lower resolution and smaller size images (thus, forming a pyramid). Hand-engineered features are then extracted from each layer in the pyramid to detect the objects, which makes the pyramid scale-invariant.--, in line 31, page 64 through line 35, page 65; and, -- applying these mobile models to object detection in a novel framework referred to as SSDLite, and further demonstrates how to build mobile semantic segmentation models through a reduced form of DeepLabv3 (referred to as Mobile DeepLabv3), is based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers. The intermediate expansion layer uses lightweight depth-wise convolutions to filter features as a source of non-linearity. The scheme allows for decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis.--, in line 10, page 70 through line 8, page 71; also see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, -- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; also see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]); and
process the obtained set of multimedia data feeds based on the determined plurality of patterns corresponding to the identified event of interest (see HASKIN: e.g., -- Streaming media is multimedia that is constantly received by and presented to an end-user while being delivered by a provider. … Live streaming refers to content delivered live over the Internet, and requires a form of source media (e.g. a video camera, an audio interface, screen capture software), an encoder to digitize the content, a media publisher, and a content delivery network to distribute and deliver the content. Streaming content may be according to, compatible with, or based on, IETF RFC 2550 entitled: “RTP: A Transport Protocol for Real-Time Applications”, IETF RFC 4587 entitled: “RTP Payload Format for H.261 Video Streams”, or IETF RFC 2326 entitled: “Real Time Streaming Protocol (RTSP)”--, in lines 10-21, page 43; also see: -- The detection mechanism is typically an electrical circuit including a re-triggerable multivibrator or another functionally similar component. Image. A digital image is a numeric representation (normally binary) of a two- dimensional image. Depending on whether the image resolution is fixed, it may be of a vector or raster type. Raster images have a finite set of digital values, called picture elements or pixels. The digital image contains a fixed number of rows and columns of pixels, which are the smallest individual element in an image, holding quantized values that represent the brightness of a given color at any specific point. Typically, the pixels are stored in computer memory as a raster image or raster map, a two-dimensional array of small integers, where these values are commonly transmitted or stored in a compressed form. The raster images can be created by a variety of input devices and techniques, such as digital cameras, scanners, coordinate-measuring machines, seismographic profiling, airborne radar, and more.--, in lines 1-11, page 11; and, -- Video content analysis transforms the audio and image stream into a set of semantically meaningful representations. The ultimate goal is to extract structural and semantic content automatically, without any human intervention, at least for limited types of video domains. Algorithms to perform content analysis include those for detecting objects in video, recognizing specific objects, persons, locations, detecting dynamic events in video, associating keywords with image regions or motion. VCA is used in a wide range of domains including entertainment, health-care, retail, automotive, transport, home automation, flame and smoke detection, safety and security. The algorithms can be implemented as software on general purpose machines, or as hardware in specialized video processing units. Many different functionalities can be implemented in VCA. Video Motion Detection is one of the simpler forms where motion is detected with regard to a fixed background scene. More advanced functionalities include video tracking and egomotion estimation. Based on the internal representation that VCA generates in the machine, it is possible to build other functionalities, such as identification, behavior analysis or other forms of situation awareness. VCA typically relies on good input video, so it is commonly combined with video enhancement technologies such as video denoising, image stabilization, unsharp masking and super- resolution--, in line 27, page 31 through line 15, page 32; also see: --YOLO is implemented as a CNN…. The YOLO approach to object detection describing frame object detection as a regression problem to spatially separated bounding boxes and associated class probabilities is described in an article authored by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi, published 9 May 2016 and entitled: “You Only Look Once: Unified, Real-Time Object Detection”, which is incorporated in its entirety for all purposes as if fully set forth herein. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance. The base YOLO model processes images in real- time at 45 frames per second while a smaller version of the network, Fast YOLO, processes an astounding 155 frames per second while still achieving double the mAP of other real-time detectors.--, in lines 10-32, page 60; --As for object detection accuracy, it achieves a mean Average Precision (mAP) of 64.16% for PASCAL VOC 2007 dataset that is only 2.63% lower than the mAP of the same YOLO network with full precision. R-CNN. Regions with CNN features (R-CNN) family is a family of machine learning models used to bypass the problem of selecting a huge number of regions. The R-CNN uses selective search to extract just 2000 regions from the image, referred to as region proposals. Then, instead of trying to classify a huge number of regions, only 2000 regions are handled. These 2000 region proposals are generated using a selective search algorithm, that includes Generating initial sub-segmentation for generating many candidate regions, using greedy algorithm to recursively combine similar regions into larger ones, and using the generated regions to produce the final candidate region proposals. These 2000 candidate region proposals are warped into a square and fed into a convolutional neural network that produces a 4096- dimensional feature vector as output.—, in lines 15-28, page 62; and,
--As in the original R-CNN, the Fast R-CNN uses Selective Search to generate its region proposals. While Fast R-CNN used Selective Search to generate ROIs, Faster R-CNN integrates the ROI generation into the neural network itself. Mask R-CNN adds instance segmentation, and also replaced ROIPooling with a new method called ROIAlign, which can represent fractions of a pixel, and Mesh R-CNN adds the ability to generate a 3D mesh from a 2D image. R-CNN and Fast R-CNN are primarily image classifier networks which are used for object detection by using Region Proposal method to generate potential bounding boxes in an image, run the classifier on these boxes, and after classification, perform post processing to tighten the boundaries of the bounding boxes and remove duplicates. Regions with CNN features (R-CNN) that combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost--, in lines 15-28, page 63;
-- A Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals, is described in an article authored by Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, published 2015, entitled: “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal networks”, which is incorporated in its entirety for all purposes as if fully set forth herein. State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet and Fast R-CNN have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. An RPN is a fully- convolutional network that simultaneously predicts object bounds and objectness scores at each position. RPNs are trained end-to-end to generate high quality region proposals, which are used by Fast R-CNN for detection. With a simple alternating optimization, RPN and Fast R-CNN can be trained to share convolutional features. For the very deep VGG-16 model, a described detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state- of-the-art object detection accuracy on PASCAL VOC 2007 (73.2% mAP) and 2012 (70.4% mAP) using 300 proposals per image. Code is available at https://github.com/ShaoqingRen/faster_rcnn. RetinaNet. RetinaNet is one of the one-stage object detection models that has proven to work well with dense and small-scale objects, that has become a popular object detection model to be used with aerial and satellite imagery. RetinaNet has been formed by making two improvements over existing single stage object detection models - Feature Pyramid Networks (FPN) and Focal Loss. Traditionally, in computer vision, featurized image pyramids have been used to detect objects with varying scales in an image. Featurized image pyramids are feature pyramids built upon image pyramids, where an image is subsampled into lower resolution and smaller size images (thus, forming a pyramid). Hand-engineered features are then extracted from each layer in the pyramid to detect the objects, which makes the pyramid scale-invariant.--, in line 31, page 64 through line 35, page 65; and, -- applying these mobile models to object detection in a novel framework referred to as SSDLite, and further demonstrates how to build mobile semantic segmentation models through a reduced form of DeepLabv3 (referred to as Mobile DeepLabv3), is based on an inverted residual structure where the shortcut connections are between the thin bottleneck layers. The intermediate expansion layer uses lightweight depth-wise convolutions to filter features as a source of non-linearity. The scheme allows for decoupling of the input/output domains from the expressiveness of the transformation, which provides a convenient framework for further analysis.--, in line 10, page 70 through line 8, page 71; also see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, -- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; also see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]).
Re Claim 7, HASKIN as modified by BRUNNER and LIN further disclose wherein to generate the output representation of the real-time event by encoding the sequenced set of multimedia data feeds and the obtained at least one input prompt using the trained vision encoder model (see HASKIN: e.g., -- Streaming media is multimedia that is constantly received by and presented to an end-user while being delivered by a provider. … Live streaming refers to content delivered live over the Internet, and requires a form of source media (e.g. a video camera, an audio interface, screen capture software), an encoder to digitize the content, a media publisher, and a content delivery network to distribute and deliver the content. Streaming content may be according to, compatible with, or based on, IETF RFC 2550 entitled: “RTP: A Transport Protocol for Real-Time Applications”, IETF RFC 4587 entitled: “RTP Payload Format for H.261 Video Streams”, or IETF RFC 2326 entitled: “Real Time Streaming Protocol (RTSP)”--, in lines 10-21, page 43; also see LIN: e.g., -- The length information can be obtained from the user. For example, it may send a prompt for asking the user to generate a long captioning information or a short captioning information, the user can give corresponding instructions according to the prompt. The length information can also be obtained by analyzing the video. When the video is a real-time captured video, it may determine the current application scenario by analyzing the video, and different length information can be determined for different application scenarios.--, in [0155], and, -- When the captioning information is generated by performing encoding, it may also generate a text caption of the video that better meets the requirements of the user by acquiring information about the captioning information that the user expects to generate. For example, during the user driving, it may collect real-time video in front of the user's line of sight and analyze the video, so as to give the user a corresponding reminder by analyzing the generated captioning information of the video or play the captioning information to the user when the user needs to be prompted, such as when there is a potential danger ahead.--, in [0366]), the processor is configured to:
encode the sequenced set of multimedia data feeds using a computer vision encoder layer of the trained vision encoder model (see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]; and, -- [0097] Compared with the spatial scene graph, the spatial-temporal scene graph adds the relationship between objects (i.e., targets) in the temporal dimension, which can better describe the spatial-temporal information of the video. In addition, the spatial-temporal scene graph can further contain the action information of the target corresponding to the temporal edge (which will be described hereinafter) to improve the accuracy of the action captioning.--, in [0097], and [0113]-[0116]; and, -- FIG. 31 illustrates a schematic flowchart of a method for generating video captioning information of the present disclosure. The 3D visual feature encoder (i.e., spatial-temporal feature extraction network), regional encoder, and semantic encoder (i.e., the semantic prediction network) shown in the figure respectively are encoders for extracting local visual features (local features shown in the figure), spatial-temporal visual features, and semantic features of respective target regions in each frame of video. Based on the local visual features, a spatial-temporal scene graph of each frame of the images can be constructed. Then, the graph convolution features (the updated local features shown in the figure) can be obtained through the graph convolution network. In this example, the 3D visual feature encoder can also be used to extract the spatial-temporal visual features of the video, and the semantic encoder can be used to extract the semantic features of the video. For the obtained spatial-temporal visual features, semantic features and graph convolution features, the feature selection network performs feature selection on various features, that is, determines the weight of each feature.--, in [0367]);
encode the obtained at least one input prompt using a word embedding layer of the trained vision encoder model (see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]; and, -- [0097] Compared with the spatial scene graph, the spatial-temporal scene graph adds the relationship between objects (i.e., targets) in the temporal dimension, which can better describe the spatial-temporal information of the video. In addition, the spatial-temporal scene graph can further contain the action information of the target corresponding to the temporal edge (which will be described hereinafter) to improve the accuracy of the action captioning.--, in [0097], and [0113]-[0116]; and, -- FIG. 31 illustrates a schematic flowchart of a method for generating video captioning information of the present disclosure. The 3D visual feature encoder (i.e., spatial-temporal feature extraction network), regional encoder, and semantic encoder (i.e., the semantic prediction network) shown in the figure respectively are encoders for extracting local visual features (local features shown in the figure), spatial-temporal visual features, and semantic features of respective target regions in each frame of video. Based on the local visual features, a spatial-temporal scene graph of each frame of the images can be constructed. Then, the graph convolution features (the updated local features shown in the figure) can be obtained through the graph convolution network. In this example, the 3D visual feature encoder can also be used to extract the spatial-temporal visual features of the video, and the semantic encoder can be used to extract the semantic features of the video. For the obtained spatial-temporal visual features, semantic features and graph convolution features, the feature selection network performs feature selection on various features, that is, determines the weight of each feature.--, in [0367]);
correlate the encoded set of multimedia data feeds with the obtained at least one input 25 prompt to identify an action of interest (see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]; and, -- [0097] Compared with the spatial scene graph, the spatial-temporal scene graph adds the relationship between objects (i.e., targets) in the temporal dimension, which can better describe the spatial-temporal information of the video. In addition, the spatial-temporal scene graph can further contain the action information of the target corresponding to the temporal edge (which will be described hereinafter) to improve the accuracy of the action captioning.--, in [0097], and [0113]-[0116]; and, -- FIG. 31 illustrates a schematic flowchart of a method for generating video captioning information of the present disclosure. The 3D visual feature encoder (i.e., spatial-temporal feature extraction network), regional encoder, and semantic encoder (i.e., the semantic prediction network) shown in the figure respectively are encoders for extracting local visual features (local features shown in the figure), spatial-temporal visual features, and semantic features of respective target regions in each frame of video. Based on the local visual features, a spatial-temporal scene graph of each frame of the images can be constructed. Then, the graph convolution features (the updated local features shown in the figure) can be obtained through the graph convolution network. In this example, the 3D visual feature encoder can also be used to extract the spatial-temporal visual features of the video, and the semantic encoder can be used to extract the semantic features of the video. For the obtained spatial-temporal visual features, semantic features and graph convolution features, the feature selection network performs feature selection on various features, that is, determines the weight of each feature.--, in [0367]); and
generate the output representation of the real-time event based on the correlation, wherein the output representation indicates the action of interest (see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]; and, -- [0097] Compared with the spatial scene graph, the spatial-temporal scene graph adds the relationship between objects (i.e., targets) in the temporal dimension, which can better describe the spatial-temporal information of the video. In addition, the spatial-temporal scene graph can further contain the action information of the target corresponding to the temporal edge (which will be described hereinafter) to improve the accuracy of the action captioning.--, in [0097], and [0113]-[0116]; and, -- When the captioning information is generated by performing encoding, it may also generate a text caption of the video that better meets the requirements of the user by acquiring information about the captioning information that the user expects to generate. For example, during the user driving, it may collect real-time video in front of the user's line of sight and analyze the video, so as to give the user a corresponding reminder by analyzing the generated captioning information of the video or play the captioning information to the user when the user needs to be prompted, such as when there is a potential danger ahead.--, in [0366]; -- FIG. 31 illustrates a schematic flowchart of a method for generating video captioning information of the present disclosure. The 3D visual feature encoder (i.e., spatial-temporal feature extraction network), regional encoder, and semantic encoder (i.e., the semantic prediction network) shown in the figure respectively are encoders for extracting local visual features (local features shown in the figure), spatial-temporal visual features, and semantic features of respective target regions in each frame of video. Based on the local visual features, a spatial-temporal scene graph of each frame of the images can be constructed. Then, the graph convolution features (the updated local features shown in the figure) can be obtained through the graph convolution network. In this example, the 3D visual feature encoder can also be used to extract the spatial-temporal visual features of the video, and the semantic encoder can be used to extract the semantic features of the video. For the obtained spatial-temporal visual features, semantic features and graph convolution features, the feature selection network performs feature selection on various features, that is, determines the weight of each feature.--, in [0367]).
Re Claim 8, HASKIN as modified by BRUNNER and LIN further disclose wherein to predict the at least one action performed in the generated output representation using the action prediction model (see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]; and, -- [0097] Compared with the spatial scene graph, the spatial-temporal scene graph adds the relationship between objects (i.e., targets) in the temporal dimension, which can better describe the spatial-temporal information of the video. In addition, the spatial-temporal scene graph can further contain the action information of the target corresponding to the temporal edge (which will be described hereinafter) to improve the accuracy of the action captioning.--, in [0097], and [0113]-[0116]; and, -- When the captioning information is generated by performing encoding, it may also generate a text caption of the video that better meets the requirements of the user by acquiring information about the captioning information that the user expects to generate. For example, during the user driving, it may collect real-time video in front of the user's line of sight and analyze the video, so as to give the user a corresponding reminder by analyzing the generated captioning information of the video or play the captioning information to the user when the user needs to be prompted, such as when there is a potential danger ahead.--, in [0366]; -- FIG. 31 illustrates a schematic flowchart of a method for generating video captioning information of the present disclosure. The 3D visual feature encoder (i.e., spatial-temporal feature extraction network), regional encoder, and semantic encoder (i.e., the semantic prediction network) shown in the figure respectively are encoders for extracting local visual features (local features shown in the figure), spatial-temporal visual features, and semantic features of respective target regions in each frame of video. Based on the local visual features, a spatial-temporal scene graph of each frame of the images can be constructed. Then, the graph convolution features (the updated local features shown in the figure) can be obtained through the graph convolution network. In this example, the 3D visual feature encoder can also be used to extract the spatial-temporal visual features of the video, and the semantic encoder can be used to extract the semantic features of the video. For the obtained spatial-temporal visual features, semantic features and graph convolution features, the feature selection network performs feature selection on various features, that is, determines the weight of each feature.--, in [0367]), the processor is configured to:
identify at least one of a type of objects, a position of objects, gestures performed within the obtained set of multimedia data feeds using a computer vision model (see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, -- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; also see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]);
classify the set of multimedia data feeds into domain specific events based on the at least one of the type of the objects, the position of the objects, and the gestures performed (see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, -- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; also see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]);
generate a confidence score for each of the classified set of multimedia data feeds using the action prediction model (see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, --[0143] Machine-learning models can be trained in a supervised manner to identify predetermined states (e.g., rearing, grooming, locomotion, immobility, freezing, or the like). The confidence of a model can be expressed in terms of a score. Such models can use sensor fusion to identify states. For example, visual data can be combined with EEG or temperature data to identify when a subject is in a sleep state. Machine learning models can also be trained to identify states in a non-supervised manner. For example, increased circling behavior and high temperature can be caused by a drug such as MK801. Such unusual behavior can be discovered with anomaly detection algorithms or other techniques. Such states are not predetermined and can lack predetermined labels or names.--, in [0143]; and,
-- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; also see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]); and
predict the at least one action performed in the generated output representation using the generated confidence score (see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, --[0143] Machine-learning models can be trained in a supervised manner to identify predetermined states (e.g., rearing, grooming, locomotion, immobility, freezing, or the like). The confidence of a model can be expressed in terms of a score. Such models can use sensor fusion to identify states. For example, visual data can be combined with EEG or temperature data to identify when a subject is in a sleep state. Machine learning models can also be trained to identify states in a non-supervised manner. For example, increased circling behavior and high temperature can be caused by a drug such as MK801. Such unusual behavior can be discovered with anomaly detection algorithms or other techniques. Such states are not predetermined and can lack predetermined labels or names.--, in [0143]; and,
-- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; also see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]).
Re Claim 9, HASKIN as modified by BRUNNER and LIN further disclose determine at one pattern with an object within the obtained set of multimedia data feeds using the trained vision encoder model (see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, --[0143] Machine-learning models can be trained in a supervised manner to identify predetermined states (e.g., rearing, grooming, locomotion, immobility, freezing, or the like). The confidence of a model can be expressed in terms of a score. Such models can use sensor fusion to identify states. For example, visual data can be combined with EEG or temperature data to identify when a subject is in a sleep state. Machine learning models can also be trained to identify states in a non-supervised manner. For example, increased circling behavior and high temperature can be caused by a drug such as MK801. Such unusual behavior can be discovered with anomaly detection algorithms or other techniques. Such states are not predetermined and can lack predetermined labels or names.--, in [0143]; and,
-- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; also see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]; -- When the captioning information is generated by performing encoding, it may also generate a text caption of the video that better meets the requirements of the user by acquiring information about the captioning information that the user expects to generate. For example, during the user driving, it may collect real-time video in front of the user's line of sight and analyze the video, so as to give the user a corresponding reminder by analyzing the generated captioning information of the video or play the captioning information to the user when the user needs to be prompted, such as when there is a potential danger ahead.--, in [0366]; -- FIG. 31 illustrates a schematic flowchart of a method for generating video captioning information of the present disclosure. The 3D visual feature encoder (i.e., spatial-temporal feature extraction network), regional encoder, and semantic encoder (i.e., the semantic prediction network) shown in the figure respectively are encoders for extracting local visual features (local features shown in the figure), spatial-temporal visual features, and semantic features of respective target regions in each frame of video. Based on the local visual features, a spatial-temporal scene graph of each frame of the images can be constructed. Then, the graph convolution features (the updated local features shown in the figure) can be obtained through the graph convolution network. In this example, the 3D visual feature encoder can also be used to extract the spatial-temporal visual features of the video, and the semantic encoder can be used to extract the semantic features of the video. For the obtained spatial-temporal visual features, semantic features and graph convolution features, the feature selection network performs feature selection on various features, that is, determines the weight of each feature.--, in [0367]); and
detect a state of the object based on the determined at least one pattern, wherein the state of the object comprises one of a mental state and a physical state of the object (see BRUNNER: e.g., -- a behavioral platform can be configured to acquire 100, 500, 1000, 2000, or more high-level features, and over about 1 million, such as over about 2 million or about 10 million, instant features. In some embodiments, a behavioral platform can obtain behavioral data concerning locomotion, trajectory complexity, body posture and shape, simple behaviors, behavioral sequences, or any combination thereof. As an additional example a behavioral platform can employ computer vision techniques to extract behavioral features concerning gait (e.g., gait geometry and gait dynamics such as stance, swing, propulsion, or the like) from image data. Such behavioral features can be used, for example, in rodent models of neurological disorders, pain and neuropathies. As described herein, such computer vision techniques can also be used to extract features representing non-gait behaviors. [0119] Consistent with disclosed embodiments, a behavioral platform can acquire observational data concerning a subject. The observational data can include subject data that concerns the subject directly, such as behavioral data or physiological data, and external data that concerns the environment in which the subject exists (e.g., whether a stimulus has been applied to the subject). The behavioral platform can extract observational features from the observational data. These observational features can include subject features extracted from subject data (e.g., behavioral features extracted from behavioral data, physiological features extracted from physiological data, or the like), or external features extracted from external data (e.g., a Boolean variable indicating application of an electric shock or flashing of a strobe light). Higher-level features can be generated from lower-level features (of the same type, e.g., a set or sequence of multiple behavioral features having a particular temporal relationship) or differing types (e.g., a combination of behavioral and physiological features indicating a state, like fear or cold). High-level features can also be generated from lower- level features in combination with observational data. Consistent with disclosed embodiments, the observational features can be used for various analyses as detailed herein.--, in [0117]-[0119]; and, --[0143] Machine-learning models can be trained in a supervised manner to identify predetermined states (e.g., rearing, grooming, locomotion, immobility, freezing, or the like). The confidence of a model can be expressed in terms of a score. Such models can use sensor fusion to identify states. For example, visual data can be combined with EEG or temperature data to identify when a subject is in a sleep state. Machine learning models can also be trained to identify states in a non-supervised manner. For example, increased circling behavior and high temperature can be caused by a drug such as MK801. Such unusual behavior can be discovered with anomaly detection algorithms or other techniques. Such states are not predetermined and can lack predetermined labels or names.--, in [0143]; and,
-- a motif can be a particular set or sequence of values (e.g., at least two time samples) in a time series stream that occurs with a probability higher than chance. Values can represent instant features and/or states. Transitions between every pair of discrete states can be from one instant feature to another, or also to the same feature. This can be a “first order” motif. In some instances, a set or sequence of several states can occur with a certain probability. Such a set or sequence of “n” states can be labeled an “n-order” motif. [0145] In some embodiments, a motif can be defined a priori and identified in a time series, or a motif can be discovered using unsupervised machine learning methods. As described herein, the time series can include observational data. This observational data can include subject data (e.g., behavioral data, physiological data, or the like) and/or external data (e.g., environmental data, indications of stimuli or rewards presented to the subject, or the like). In some embodiments, a times series can include multiple types of data (e.g., different types of subject data, or a combination of subject data and external data, or the like).--, in [0144]-[0145]; also see LIN: e.g., -- [0071] Specifically, the trained semantic prediction network can be used to obtain the semantic features of the entire video or image. As an example, a structure diagram of a semantic prediction network is shown in FIG. 6. As shown in the figure, the semantic prediction network may include a CNN and a multi-classification structure (multi-classification shown in the figure). Taking the video shown in FIG. 6 as an example, when using this network, frames of the selected video are input to the semantic prediction network, the video features are extracted from each frame by using CNN, and a multi-classification operation is performed on the extracted video features through a multi-classification structure to obtain the probabilities corresponding to multiple predefined semantic features of the video, and finally output one or more predefined semantic features among the multiple predefined semantic features according to the probabilities. As shown in FIG. 6, based on the input frames (the images including person and dogs shown in the figure), the probabilities corresponding to various semantic features, such as person, dogs and roads, can be obtained through the semantic prediction network, for example, as shown in figure, the probability of including person is 0.95, and the probability of including dog is 0.8. Based on the predicted probabilities and the pre-configured semantic feature filtering rules, the semantic features with probabilities being greater than a set threshold can be output, or the set number of semantic features with higher probability are output.--, in [0071]; and, -- [0089] In the traditional way of constructing the spatial scene graph, the attributes of objects (i.e., targets) are not distinguished, and various attributes are classified by a classifier, so the accuracy of the obtained attributes is low. However, when the spatial scene graph is constructed based on the attribute features predicted by this scheme, it can obtain more specific target attribute information, including noun attributes (such as clothes, glasses, etc.), verb attributes (such as standing, walking, etc.), adjective attributes (such as tall, fat, etc.), and relatively rare attributes (such as blue whale, pony, etc.). In addition, different classifiers can be respectively used to obtain the different types of attributes. As such, the obtained attributes are more accurate and the attributes are more diversified, thereby generating more accurate captioning information based on the predicted attribute features.--, in [0089]; and, -- When the captioning information is generated by performing encoding, it may also generate a text caption of the video that better meets the requirements of the user by acquiring information about the captioning information that the user expects to generate. For example, during the user driving, it may collect real-time video in front of the user's line of sight and analyze the video, so as to give the user a corresponding reminder by analyzing the generated captioning information of the video or play the captioning information to the user when the user needs to be prompted, such as when there is a potential danger ahead.--, in [0366]; -- FIG. 31 illustrates a schematic flowchart of a method for generating video captioning information of the present disclosure. The 3D visual feature encoder (i.e., spatial-temporal feature extraction network), regional encoder, and semantic encoder (i.e., the semantic prediction network) shown in the figure respectively are encoders for extracting local visual features (local features shown in the figure), spatial-temporal visual features, and semantic features of respective target regions in each frame of video. Based on the local visual features, a spatial-temporal scene graph of each frame of the images can be constructed. Then, the graph convolution features (the updated local features shown in the figure) can be obtained through the graph convolution network. In this example, the 3D visual feature encoder can also be used to extract the spatial-temporal visual features of the video, and the semantic encoder can be used to extract the semantic features of the video. For the obtained spatial-temporal visual features, semantic features and graph convolution features, the feature selection network performs feature selection on various features, that is, determines the weight of each feature.--, in [0367]).
Re Claims 10-18, claim 10-18 are the corresponding method claim to claims 1-9 respectively. Thus, claims 10-18 are rejected for the similar reasons as for claims 1-9. Furthermore, HASKIN as modified by BRUNNER and LIN further disclose method comprising: obtaining, by a processor, a set of multimedia data feeds from a plurality of image capturing devices, wherein the set of multimedia data feeds correspond to a real-time event, and wherein the set of multimedia data feeds correspond to a time-series data captured at a plurality of time intervals and a multi-resolution data captured from the plurality of image capturing devices; processing, by the processor, the obtained set of multimedia data feeds (see HASKIN: e.g., Fig. 1, and Fig. 8; --An image processor block 13 receives the analog signal from the image sensor 12. The Analog Front End (AFE) in the block 13 filters, amplifies, and digitizes the signal, using an analog-to-digital (A/D) converter. The AFE further provides Correlated Double Sampling (CDS) and provides a gain control to accommodate varying illumination conditions. In the case of a CCD-based sensor 12, a CCD AFE (Analog Front End) component may be used between the digital image processor 13 and the sensor 12. Such an AFE may be based on VSP2560 ‘CCD Analog Front End for Digital Cameras’ available from Texas Instruments Incorporated of Dallas, Texas, U.S.A. The block 13 further contains a digital image processor, which receives the digital data from the AFE, and processes this digital representation of the image to handle various industry standards, and executes various computations and algorithms. Preferably, additional image enhancements may be performed by the block 13 such as generating greater pixel density or adjusting color balance, contrast, and luminance. Further, the block 13 may perform other data management functions and processing on the raw digital image data. Commonly, the timing relationship of the vertical / horizontal reference signals and the pixel clock are also handled in this block. Digital Media System-on-Chip device TMS320DM357 available from Texas Instruments Incorporated of Dallas, Texas, U.S.A. is an example of a device implementing in a single chip (and associated circuitry) part or all of the image processor 13, part or all of a video compressor 14 and part or all of a transceiver 15.--, in Description section, lines 10-35, page 2; and, -- Streaming media is multimedia that is constantly received by and presented to an end-user while being delivered by a provider. … Live streaming refers to content delivered live over the Internet, and requires a form of source media (e.g. a video camera, an audio interface, screen capture software), an encoder to digitize the content, a media publisher, and a content delivery network to distribute and deliver the content. Streaming content may be according to, compatible with, or based on, IETF RFC 2550 entitled: “RTP: A Transport Protocol for Real-Time Applications”, IETF RFC 4587 entitled: “RTP Payload Format for H.261 Video Streams”, or IETF RFC 2326 entitled: “Real Time Streaming Protocol (RTSP)”--, in lines 10-21, page 43; also see: -- The detection mechanism is typically an electrical circuit including a re-triggerable multivibrator or another functionally similar component. Image. A digital image is a numeric representation (normally binary) of a two- dimensional image. Depending on whether the image resolution is fixed, it may be of a vector or raster type. Raster images have a finite set of digital values, called picture elements or pixels. The digital image contains a fixed number of rows and columns of pixels, which are the smallest individual element in an image, holding quantized values that represent the brightness of a given color at any specific point. Typically, the pixels are stored in computer memory as a raster image or raster map, a two-dimensional array of small integers, where these values are commonly transmitted or stored in a compressed form. The raster images can be created by a variety of input devices and techniques, such as digital cameras, scanners, coordinate-measuring machines, seismographic profiling, airborne radar, and more.--, in lines 1-11, page 11; and, -- Video content analysis transforms the audio and image stream into a set of semantically meaningful representations. The ultimate goal is to extract structural and semantic content automatically, without any human intervention, at least for limited types of video domains. Algorithms to perform content analysis include those for detecting objects in video, recognizing specific objects, persons, locations, detecting dynamic events in video, associating keywords with image regions or motion. VCA is used in a wide range of domains including entertainment, health-care, retail, automotive, transport, home automation, flame and smoke detection, safety and security. The algorithms can be implemented as software on general purpose machines, or as hardware in specialized video processing units. Many different functionalities can be implemented in VCA. Video Motion Detection is one of the simpler forms where motion is detected with regard to a fixed background scene. More advanced functionalities include video tracking and egomotion estimation. Based on the internal representation that VCA generates in the machine, it is possible to build other functionalities, such as identification, behavior analysis or other forms of situation awareness. VCA typically relies on good input video, so it is commonly combined with video enhancement technologies such as video denoising, image stabilization, unsharp masking and super- resolution--, in line 27, page 31 through line 15, page 32; also see: --YOLO is implemented as a CNN…. The YOLO approach to object detection describing frame object detection as a regression problem to spatially separated bounding boxes and associated class probabilities is described in an article authored by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi, published 9 May 2016 and entitled: “You Only Look Once: Unified, Real-Time Object Detection”, which is incorporated in its entirety for all purposes as if fully set forth herein. A single neural network predicts bounding boxes and class probabilities directly from full images in one evaluation. Since the whole detection pipeline is a single network, it can be optimized end-to-end directly on detection performance. The base YOLO model processes images in real- time at 45 frames per second while a smaller version of the network, Fast YOLO, processes an astounding 155 frames per second while still achieving double the mAP of other real-time detectors.--, in lines 10-32, page 60).
Re Claim 19-20, claim 19-20 are the corresponding medium claim to claims 1, and 4 respectively. Thus, claims 19-20 are rejected for the similar reasons as for claims 1, and 4. Furthermore, HASKIN as modified by BRUNNER and LIN further disclose a non-transitory computer readable medium comprising a processor-executable instructions that cause a processor to perform the steps (see HASKIN: e.g., Fig. 1, and Fig. 8; --An image processor block 13 receives the analog signal from the image sensor 12. The Analog Front End (AFE) in the block 13 filters, amplifies, and digitizes the signal, using an analog-to-digital (A/D) converter. The AFE further provides Correlated Double Sampling (CDS) and provides a gain control to accommodate varying illumination conditions. In the case of a CCD-based sensor 12, a CCD AFE (Analog Front End) component may be used between the digital image processor 13 and the sensor 12. Such an AFE may be based on VSP2560 ‘CCD Analog Front End for Digital Cameras’ available from Texas Instruments Incorporated of Dallas, Texas, U.S.A. The block 13 further contains a digital image processor, which receives the digital data from the AFE, and processes this digital representation of the image to handle various industry standards, and executes various computations and algorithms. Preferably, additional image enhancements may be performed by the block 13 such as generating greater pixel density or adjusting color balance, contrast, and luminance. Further, the block 13 may perform other data management functions and processing on the raw digital image data. Commonly, the timing relationship of the vertical / horizontal reference signals and the pixel clock are also handled in this block. Digital Media System-on-Chip device TMS320DM357 available from Texas Instruments Incorporated of Dallas, Texas, U.S.A. is an example of a device implementing in a single chip (and associated circuitry) part or all of the image processor 13, part or all of a video compressor 14 and part or all of a transceiver 15.--, in Description section, lines 10-35, page 2).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEI WEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on 8:00 - 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662
Prosecution Insights