![\"Elastic](\"https:\/\/cdn.services-k8s.prod.aws.htb.systems\/content\/modules\/211\/elastic.png\")
<\/figure>\n\n\n\nThe high-level architecture of the Elastic stack can be enhanced in resource-intensive environments with the addition of Kafka, RabbitMQ, and Redis for buffering and resiliency, and nginx for security.<\/p>\n\n\n\n Let’s delve into each component of the Elastic stack.<\/p>\n\n\n\n Note: The Elastic stack can be used as a Security Information and Event Management (SIEM) solution to collect, store, analyze, and visualize security-related data from various sources.<\/p>\n\n\n\n To implement the Elastic stack as a SIEM solution, security-related data from various sources such as firewalls, IDS\/IPS, and endpoints should be ingested into the Elastic stack using Logstash. Elasticsearch should be configured to store and index the security data, and Kibana should be used to create custom dashboards and visualizations to provide insights into security-related events.<\/p>\n\n\n\n To detect security-related incidents, Elasticsearch can be used to perform searches and correlations on the collected security data.<\/p>\n\n\n\n As Security Operations Center (SOC) analysts, we are likely to extensively use Kibana as our primary interface when working with the Elastic stack. Therefore, it is essential to become proficient with its functionalities and features.<\/p>\n\n\n\n Kibana Query Language (KQL) is a powerful and user-friendly query language designed specifically for searching and analyzing data in Kibana. It simplifies the process of extracting insights from your indexed Elasticsearch data, offering a more intuitive approach than Elasticsearch’s Query DSL. Let’s explore the technical aspects and key components of the KQL language.<\/p>\n\n\n\n Introduction To The Elastic Stack<\/p>\n\n\n\n The KQL query By using this query, SOC analysts can identify failed login attempts on Windows machines within the Elasticsearch index, and investigate the source of the attempts and potential security threats. This type of query can help identify brute force attacks, password guessing, and other suspicious activities related to login attempts on Windows systems.<\/p>\n\n\n\n By further refining the query with additional conditions, such as the source IP address, username, or time range, SOC analysts can gain more specific insights and effectively investigate potential security incidents.<\/p>\n\n\n\n Introduction To The Elastic Stack<\/p>\n\n\n\n This query returns records containing the string “svc-sql1” in any indexed field.<\/p>\n\n\n\n Introduction To The Elastic Stack<\/p>\n\n\n\n The KQL query In Windows, the SubStatus value indicates the reason for a login failure. A SubStatus value of 0xC0000072 indicates that the account is currently disabled.<\/p>\n\n\n\n By using this query, SOC analysts can identify failed login attempts against disabled accounts. Such a behavior requires further investigation, as the disabled account’s credentials may have been identified somehow by an attacker.<\/p>\n\n\n\n Introduction To The Elastic Stack<\/p>\n\n\n\n By using this query, SOC analysts can identify failed login attempts against disabled accounts that took place between March 3rd 2023 and March 6th 2023<\/p>\n\n\n\n Introduction To The Elastic Stack<\/p>\n\n\n\n The Kibana KQL query This query (if extended) can be useful in identifying potentially malicious login attempts targeted at administrator accounts.<\/p>\n\n\n\n “How can I identify the available fields and values?”, you may ask. Let’s see how we could have identified the available fields and values that we used in this section.<\/p>\n\n\n\n Example<\/strong>: Identify failed login attempts against disabled accounts that took place between March 3rd 2023 and March 6th 2023 KQL:<\/p>\n\n\n\n Introduction To The Elastic Stack<\/p>\n\n\n\n Data and field identification approach 1: Leverage KQL’s free text search<\/strong><\/p>\n\n\n\n Using the Discover<\/a> feature, we can effortlessly explore and sift through the available data, as well as gain insights into the architecture of the available fields, before we start constructing KQL queries.<\/p>\n\n\n\n Data and field identification approach 2: Leverage Elastic’s documentation<\/strong><\/p>\n\n\n\n It could be a good idea to first familiarize ourselves with Elastic’s comprehensive documentation before delving into the “Discover” feature. The documentation provides a wealth of information on the different types of fields we may encounter. Some good resources to start with are:<\/p>\n\n\n\n Elastic Common Schema (ECS) is a shared and extensible vocabulary for events and logs across the Elastic Stack, which ensures consistent field formats across different data sources. When it comes to Kibana Query Language (KQL) searches within the Elastic Stack, using ECS fields presents several key advantages:<\/p>\n\n\n\n What Is The Elastic Stack? The Elastic stack, created by Elastic, is an open-source collection of mainly three applications (Elasticsearch, Logstash, and Kibana) that work in harmony to offer users comprehensive search and visualization capabilities for real-time analysis and exploration of log file sources. The high-level architecture of the Elastic stack can be enhanced in…<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"doc_category":[46],"doc_tag":[],"class_list":["post-7968","docs","type-docs","status-publish","hentry","doc_category-security-monitoring-siem-fundamentals"],"year_month":"2026-04","word_count":1800,"total_views":0,"reactions":{"happy":0,"normal":0,"sad":0},"author_info":{"name":"admin","author_nicename":"admin","author_url":"https:\/\/scalemedia.co.za\/cybermedia\/author\/admin\/"},"doc_category_info":[{"term_name":"Security Monitoring & SIEM Fundamentals","term_url":"https:\/\/scalemedia.co.za\/cybermedia\/docs-category\/security-monitoring-siem-fundamentals\/"}],"doc_tag_info":[],"_links":{"self":[{"href":"https:\/\/scalemedia.co.za\/cybermedia\/wp-json\/wp\/v2\/docs\/7968","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scalemedia.co.za\/cybermedia\/wp-json\/wp\/v2\/docs"}],"about":[{"href":"https:\/\/scalemedia.co.za\/cybermedia\/wp-json\/wp\/v2\/types\/docs"}],"author":[{"embeddable":true,"href":"https:\/\/scalemedia.co.za\/cybermedia\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/scalemedia.co.za\/cybermedia\/wp-json\/wp\/v2\/comments?post=7968"}],"version-history":[{"count":1,"href":"https:\/\/scalemedia.co.za\/cybermedia\/wp-json\/wp\/v2\/docs\/7968\/revisions"}],"predecessor-version":[{"id":7969,"href":"https:\/\/scalemedia.co.za\/cybermedia\/wp-json\/wp\/v2\/docs\/7968\/revisions\/7969"}],"wp:attachment":[{"href":"https:\/\/scalemedia.co.za\/cybermedia\/wp-json\/wp\/v2\/media?parent=7968"}],"wp:term":[{"taxonomy":"doc_category","embeddable":true,"href":"https:\/\/scalemedia.co.za\/cybermedia\/wp-json\/wp\/v2\/doc_category?post=7968"},{"taxonomy":"doc_tag","embeddable":true,"href":"https:\/\/scalemedia.co.za\/cybermedia\/wp-json\/wp\/v2\/doc_tag?post=7968"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"Beats](\"https:\/\/cdn.services-k8s.prod.aws.htb.systems\/content\/modules\/211\/elastic1.png\")
<\/figure>\n\n\n\nElasticsearch<\/code> is a distributed and JSON-based search engine, designed with RESTful APIs. As the core component of the Elastic stack, it handles indexing, storing, and querying. Elasticsearch empowers users to conduct sophisticated queries and perform analytics operations on the log file records processed by Logstash.<\/p>\n\n\n\nLogstash<\/code> is responsible for collecting, transforming, and transporting log file records. Its strength lies in its ability to consolidate data from various sources and normalize them. Logstash operates in three main areas:<\/p>\n\n\n\n\n
Process input<\/code>: Logstash ingests log file records from remote locations, converting them into a format that machines can understand. It can receive records through different\u00a0input methods<\/a>, such as reading from a flat file, a TCP socket, or directly from syslog messages. After processing the input, Logstash proceeds to the next function.<\/li>\n\n\n\nTransform and enrich log records<\/code>: Logstash offers numerous ways to\u00a0modify a log record<\/a>‘s format and even content. Specifically, filter plugins can perform intermediary processing on an event, often based on a predefined condition. Once a log record is transformed, Logstash processes it further.<\/li>\n\n\n\nSend log records to Elasticsearch<\/code>: Logstash utilizes\u00a0output plugins<\/a>\u00a0to transmit log records to Elasticsearch.<\/li>\n<\/ol>\n\n\n\nKibana<\/code> serves as the visualization tool for Elasticsearch documents. Users can view the data stored in Elasticsearch and execute queries through Kibana. Additionally, Kibana simplifies the comprehension of query results using tables, charts, and custom dashboards.<\/p>\n\n\n\nBeats<\/code> is an additional component of the Elastic stack. These lightweight, single-purpose data shippers are designed to be installed on remote machines to forward logs and metrics to either Logstash or Elasticsearch directly. Beats simplify the process of collecting data from various sources and ensure that the Elastic Stack receives the necessary information for analysis and visualization.<\/p>\n\n\n\nBeats<\/code> -> Logstash<\/code> -> Elasticsearch<\/code> -> Kibana<\/code><\/p>\n\n\n\n![\"Beats](\"https:\/\/cdn.services-k8s.prod.aws.htb.systems\/content\/modules\/211\/beats1.png\")
<\/figure>\n\n\n\nBeats<\/code> -> Elasticsearch<\/code> -> Kibana<\/code><\/p>\n\n\n\n![\"Beats](\"https:\/\/cdn.services-k8s.prod.aws.htb.systems\/content\/modules\/211\/beats2.png\")
<\/figure>\n\n\n\n
\n\n\n\nThe Elastic Stack As A SIEM Solution<\/h2>\n\n\n\n
![\"Elastic](\"https:\/\/cdn.services-k8s.prod.aws.htb.systems\/content\/modules\/211\/discover.png\")
<\/figure>\n\n\n\n\n
Basic Structure<\/code>: KQL queries are composed of\u00a0field:value<\/code>\u00a0pairs, with the field representing the data’s attribute and the value representing the data you’re searching for. For example:<\/li>\n<\/ul>\n\n\n\nevent.code:4625\n<\/code><\/pre>\n\n\n\nevent.code:4625<\/code> filters data in Kibana to show events that have the Windows event code 4625<\/a>. This Windows event code is associated with failed login attempts in a Windows operating system.<\/p>\n\n\n\n\n
Free Text Search<\/code>: KQL supports free text search, allowing you to search for a specific term across multiple fields without specifying a field name. For instance:<\/li>\n<\/ul>\n\n\n\n"svc-sql1"\n<\/code><\/pre>\n\n\n\n\n
Logical Operators<\/code>: KQL supports logical operators AND, OR, and NOT for constructing more complex queries. Parentheses can be used to group expressions and control the order of evaluation. For example:<\/li>\n<\/ul>\n\n\n\nevent.code:4625 AND winlog.event_data.SubStatus:0xC0000072\n<\/code><\/pre>\n\n\n\nevent.code:4625 AND winlog.event_data.SubStatus:0xC0000072<\/code> filters data in Kibana to show events that have the Windows event code 4625 (failed login attempts) and the SubStatus value of 0xC0000072.<\/p>\n\n\n\n\n
Comparison Operators<\/code>: KQL supports various comparison operators such as\u00a0:<\/code>,\u00a0:><\/code>,\u00a0:>=<\/code>,\u00a0:<<\/code>,\u00a0:<=<\/code>, and\u00a0:!<\/code>. These operators enable you to define precise conditions for matching field values. For instance:<\/li>\n<\/ul>\n\n\n\nevent.code:4625 AND winlog.event_data.SubStatus:0xC0000072 AND @timestamp >= "2023-03-03T00:00:00.000Z" AND @timestamp <= "2023-03-06T23:59:59.999Z"\n<\/code><\/pre>\n\n\n\n\n
Wildcards and Regular Expressions<\/code>: KQL supports wildcards and regular expressions to search for patterns in field values. For example:<\/li>\n<\/ul>\n\n\n\nevent.code:4625 AND user.name: admin*\n<\/code><\/pre>\n\n\n\nevent.code:4625 AND user.name: admin*<\/code> filters data in Kibana to show events that have the Windows event code 4625 (failed login attempts) and where the username starts with “admin”, such as “admin”, “administrator”, “admin123”, etc.<\/p>\n\n\n\n
\n\n\n\nHow To Identify The Available Data<\/h2>\n\n\n\n
\n\n\n\nevent.code:4625 AND winlog.event_data.SubStatus:0xC0000072 AND @timestamp >= "2023-03-03T00:00:00.000Z" AND @timestamp <= "2023-03-06T23:59:59.999Z"\n<\/code><\/pre>\n\n\n\n\n
"4625"<\/code>. In the returned records we notice\u00a0event.code:4625<\/code>,\u00a0winlog.event_id:4625<\/code>, and\u00a0@timestamp<\/code>\n\n
event.code<\/code>\u00a0is related to the\u00a0Elastic Common Schema (ECS)<\/a><\/li>\n\n\n\nwinlog.event_id<\/code>\u00a0is related to\u00a0Winlogbeat<\/a><\/li>\n\n\n\n@timestamp<\/code>\u00a0typically contains the time extracted from the original event and it is\u00a0different from\u00a0event.created<\/code><\/a>\u00a0![\"Elastic](\"https:\/\/cdn.services-k8s.prod.aws.htb.systems\/content\/modules\/211\/discover1.png\")
<\/li>\n<\/ul>\n<\/li>\n\n\n\n"0xC0000072"<\/code>. By expanding the returned record we notice\u00a0winlog.event_data.SubStatus<\/code>\u00a0that is related to\u00a0Winlogbeat<\/a>\u00a0![\"Elastic](\"https:\/\/cdn.services-k8s.prod.aws.htb.systems\/content\/modules\/211\/discover2.png\")
<\/li>\n<\/ul>\n\n\n\n\n
The Elastic Common Schema (ECS)<\/h2>\n\n\n\n
\n\n\n\n\n
Unified Data View<\/code>: ECS enforces a structured and consistent approach to data, allowing for unified views across multiple data sources. For instance, data originating from Windows logs, network traffic, endpoint events, or cloud-based data sources can all be searched and correlated using the same field names.<\/li>\n\n\n\nImproved Search Efficiency<\/code>: By standardizing the field names across different data types, ECS simplifies the process of writing queries in KQL. This means that analysts can efficiently construct queries without needing to remember specific field names for each data source.<\/li>\n\n\n\nEnhanced Correlation<\/code>: ECS allows for easier correlation of events across different sources, which is pivotal in cybersecurity investigations. For example, you can correlate an IP address involved in a security incident with network traffic logs, firewall logs, and endpoint data to gain a more comprehensive understanding of the incident.<\/li>\n\n\n\nBetter Visualizations<\/code>: Consistent field naming conventions improve the efficacy of visualizations in Kibana. As all data sources adhere to the same schema, creating dashboards and visualizations becomes easier and more intuitive. This can help in spotting trends, identifying anomalies, and visualizing security incidents.<\/li>\n\n\n\nInteroperability with Elastic Solutions<\/code>: Using ECS fields ensures full compatibility with advanced Elastic Stack features and solutions, such as Elastic Security, Elastic Observability, and Elastic Machine Learning. This allows for advanced threat hunting, anomaly detection, and performance monitoring.<\/li>\n\n\n\nFuture-proofing<\/code>: As ECS is the foundational schema across the Elastic Stack, adopting ECS ensures future compatibility with enhancements and new features that are introduced into the Elastic ecosystem.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"