SPLK-3003 Sample Questions & Answers

The MC is a search head that gathers data by searching other instances. Most instances (search heads, deployment servers, the license manager, non-clustered indexers) are added to it as search peers, but the docs say: "Do not add clustered indexers." Instead, you add the cluster manager as a search peer and configure the MC instance as a search head of the indexer cluster, as a multisite search head when the cluster is multisite. The MC then reaches the cluster's peers through the cluster, the same way any cluster search head does. If that step is missed, the cluster's indexers are missing from the MC even though the other instances appear. There is no [mc_roles] stanza, and licenses do not restrict monitoring.

False. site0 is not a valid site for a peer node. According to server.conf, site0 can be set only on search heads or on forwarders that participate in indexer discovery. On a search head it disables search affinity; on an indexer-discovery forwarder it sends data to peers on all sites. Every peer must belong to a real site (site1 to site63) so that the manager can apply the site replication and search factors.

Define one target group per destination in outputs.conf (for example [tcpout:sec_cluster] and [tcpout:ops_cluster], each with its server list). Then set _TCP_ROUTING = in each input stanza of inputs.conf, so security inputs go to sec_cluster and operations inputs go to ops_cluster. A universal forwarder can route by data input this way. Event-based routing through props.conf/transforms.conf works only on a heavy forwarder.

No extra setting is needed. Native Splunk authentication always takes precedence over external schemes, so native accounts keep working while SAML is enabled. To bypass the SAML redirect, the administrators browse to https:// : /en-US/account/login?loginType=splunk and sign in with their local credentials. This works even when the IdP is unreachable. authentication.conf has no "fallback" setting, authType accepts only one value, and [roleMap_SAML] maps IdP groups to roles; it does not enable local login.

maxConcurrentDownloads in the [global] stanza of serverclass.conf sets the maximum number of deployment clients that can download app bundles from the deployment server at the same time. A client that is refused retries at its next phone home. The default 0 means no limit, so raising the throttled value (or resetting it to 0) lets more forwarders download at once. phoneHomeIntervalInSecs is a client-side deploymentclient.conf setting. crossServerChecksum keeps app checksums consistent across several deployment servers behind a load balancer. restartSplunkd only controls whether clients restart after an app update.

If the replication_factor in server.conf is, for example, 3, but there are only 2 active peer nodes, the cluster master will be in a constant state of trying to create a third copy of every bucket. It will continuously fail to find a valid target peer, leading to an endless cycle of bucket-fixing activities and high CPU load on the master node. This is a common misconfiguration issue in undersized or partially failed clusters.

A Search Head Cluster (SHC) requires a Deployer to manage and distribute configurations (apps, conf files) to all cluster members. This ensures that every member has an identical set of configurations, which is critical for consistent behavior and functionality. The Deployer is a separate Splunk instance dedicated to this role.

The Search Job Inspector defines command.search.rawdata as the time it took to read the actual events from the rawdata files, and command.search.index as the time spent looking in the tsidx files to work out which events to retrieve. When reading rawdata takes most of the run time, the indexers are reading and decompressing a very large number of events. That usually means the base search terms do not narrow the search through the indexed terms (for example a broad search, or one that filters only on search-time field values), so most of the events read are discarded afterwards. Subsearch time is reported under dispatch.evaluate, and time the search head spends waiting for its peers under dispatch.fetch.

The fishbucket ($SPLUNK_DB/fishbucket/splunk_private_db) is the database where monitor inputs store their file checkpoints (CRC and seek address), which record how far each file has been read. btprobe (run with splunk cmd btprobe, with Splunk stopped) queries these checkpoints and can reset one file's checkpoint with --reset, which re-indexes that file. The REST endpoint /services/admin/inputstatus/TailingProcessor:FileStatus shows the live status of tailed files. These tools help troubleshoot files that are not read or are read twice.

In a Search Head Cluster, the Captain is responsible for replicating runtime changes to knowledge objects (like reports, dashboards, etc.) to all other members. The Deployer is used for baseline app configurations, not for runtime user content. The log messages on the captain, combined with the symptom of delayed visibility of user content, point directly to a bottleneck or issue with the captain's replication process. This is a core function of the captain, separate from the deployer or indexer cluster interactions.